-
I signed up for WintellectNOW and am watching Bruce Dawson’s courses on Event Tracing for Windows.
-
dbremner
1 month ago
|
link
| on:
Ask: Do you use generative testing? How would you improve your use of it?
Do you use it?
I used combinatorial testing in a project for work last year.In which kind of application?
AuditAssistant is a Windows Forms application in C# that generates software inventory reports and files them to a document management system. My employer used it for our migration to Windows 10.What framework do you use?
I used the XUnit.Combinatorial library for xUnit.Are you happy with the results?
Generative testing played a small role in this project. I used XUnit.Combinatorial for testing two DTOs. Both classes check their preconditions but neither of them contain any other logic. The results are a mixed bag. I didn’t write enough generative tests for them to really pay off. They would be more valuable in a larger project with more complex invariants to enforce.What would you like to have improved in generative testing in general?
I would like a book on generative testing along the lines of Growing Object-Oriented Software Guided by Tests.
-
I’m a primary contributor to c2rust and I may be the person “stolen” away from corrode. I’d like to apologize if it feels like we ripped off ideas without giving due credit - the project wasn’t really supposed to “discovered” so soon. The website was a throwaway idea, a means of easily sharing our work in a limited circle while avoiding both the DARPA approval and the sub-par build process (you have to build a huge chunk of Clang).
So here is me acknowledging Jamey’s work: I personally did take inspiration from Corrode, and I was expecting to work on Corrode proper when I joined Galois (I even wrote a pure Haskell library to parse/pretty-print Rust code - with Corrode in mind). I’ve re-read the CFG module of Corrode several times (as well as the Mozilla paper, and some older literature).
All that said, I also want to point out that Corrode hasn’t had any activity at all since last April - and that’s not for want of PRs piling up. I’m not criticizing here, since I understand that managing an open source can be quite time-consuming and stressful, but I feel like this also does need to be mentioned. Also, c2rust can be freely forked. Once the DARPA funding runs out, it is my hope that the Rust community will become the maintainers.
Finally, regarding the many improvements that can be automated: that is next up on our plate!
-
Hi, nice to see you here!
Regarding idiomatic improvements, in case you missed, C2Eif is the most impressive research in this area in my opinion. It can create classes and methods based on function signature analysis. It can also mark methods as private based on call graph analysis.
It is also impressive in coverage. It handles Vim, it handles inline assembly, it translates long jumps to exceptions, it handles variadic functions (Eiffel doesn’t have them like Rust, they are translated to library calls), it translates printf format strings to Eiffel formatting, etc.
-
-
-
I didn’t see it in search. That project would’ve been a good submission for Lobsters. :)
-
-
-
The pull request from David Fowl is also worth a read. It uses Spans and System.IO.Pipelines.
-
I’m going to try out the beta for Pulumi (a local startup).
-
dbremner
2 months ago
|
link
| on:
"can you explain to some degree why SPARC chips run java so well?"
This blog entry follows Betteridge’s Rule of Headlines and it isn’t worth reading.
-
I liked this article and appreciate the examples. But I was left wondering: under what conditions, if any, should I not follow the advice in this article?
-
I think it’s all good as rules of thumb, but I see two potential downsides.
First, there’s a key phrase that I think indicates some limitations:
It’s not a general purpose BMP library. It only supports 24-bit true color, ignoring most BMP features such as palettes.
As you start adding features, this design approach may rapidly become intractable. It’s most suitable for libraries that are minimalist in the other sense, and unfortunately, we live in an extremely complicated world where such libraries often aren’t actually useful. It’s all great ideals, but if taken as absolute gospel, I suspect it will strongly interfere with getting stuff done.
Second, for instance, I don’t like how the author has
bmp_sizereturn 0 in the case of an error. I know it’s a C-ism that’s very difficult to work around in C, but that’s a totally valid size to pass tocalloc, which may return a totally valid non-NULLpointer, andbmp_initmay then scribble over totally arbitrary memory. (Or it might segfault. “Might” is the operative word here. When a segfault is the best possible outcome, something has gone badly, badly wrong.) This approach forces the caller to check, which is unfriendly and error-prone. Of course, I don’t actually know a better way in C to handle it; returning a negative signed value won’t work because of conversion rules (though trying to allocate almostSIZE_MAXbytes of memory will probably fail).Similarly,
…the library returns the power of two number of uint32_t it needs to allocate….
Why? The caller will definitely forget to
1ULL << zeventually, and then they’ll have far, far too small a table for the number of elements they want to put in it (which, by the way, the library also has no way to signal). I guess it saves the library from an “error state”, but at the cost of obfuscating its own interface and pushing the check for that error state (which, of course, didn’t actually go away) into the application where it will need to be written many times and thus have many opportunities to be done incorrectly.…I guess I’m complaining that C isn’t Rust here. =/ But in both these cases, I wonder if a less “minimalist” interface might have been less error-prone to actually use, and (hopefully) uses of a library greatly outnumber implementations thereof, and thus should be a preferable target for minimization.
-
Guideline #2 (No dynamic memory allocations) can have a significant effect on error detection.
If the library allocates memory then it can use structure marking to detect use after free, uninitialized variables, and other programming errors.
Here is a quick sketch of a version that uses structure marking. I added bmp_try_bytes so that the resulting bitmap can be written to a file.
enum bmp_error_code //definition omitted for brevity struct bmp_data; //incomplete type bmp_error_code bmp_try_create(long width, long height, struct bmp_data **result); bmp_error_code bmp_try_set(struct bmp_data *, long x, long y, unsigned long color); bmp_error_code bmp_try_get(const struct bmp_data *, long x, long y, unsigned long *color); bmp_error_code bmp_try_free(struct bmp_data **); bmp_error_code bmp_try_get_bytes(const struct bmp_data *, size_t *bytes, void **bytes);
-
-
Kinda late on UNIX bashing bandwagon :)
Also, Windows owes more of it’s legacy to VMS.
-
It does, but users don’t use any of the VMSy goodness in Windows: To them it’s just another shitty UNIX clone, with everything being a file or a program (which is also a file). I think that’s the point.
Programmers rarely even use the VMSy goodness, especially if they also want their stuff to work on Mac. They treat Windows as a kind of retarded UNIX cousin (which is a shame because the API is better; IOCP et al)
Sysadmins often struggle with Windows because of all the things underneath that aren’t files.
Message/Object operating systems are interesting, but for the most part (OS/2, BeOS, QNX) they, for the most part, degraded into this “everything is a file” nonsense…
Until they got rid of the shared filesystem: iOS finally required messaging for applications to communicate on their own, and while it’s been rocky, it’s starting to paint a picture to the next generation who will finally make an operating system without files.
-
-
-
Bowels is a good metaphor. There’s good stuff in Windows, but you’ve got to put on a shoulder length glove and grab a vat of crisco before you can find any of it.
-
-
-
I think you’re being a little bit harsh. End-users definitely don’t grok the VMSy goodness; I agree. And maybe the majority of developers don’t, either (though I doubt the majority of Linux devs grok
journaldv. syslogs, really understand how to use/proc, grok Linux namespaces, etc.). But I’ve worked with enough Windows shops to promise you that a reasonable number of Windows developers do get the difference.That said, I have a half-finished book from a couple years ago, tentatively called Windows Is Not Linux, which dove into a lot of the, “okay, I know you want to do
$xbecause that’s how you did it on Linux, and doing$xon Windows stinks, so you think Windows stinks, but let me walk you through$yand explain to you why it’s at least as good as the Linux way even though it’s different,” specifically because I got fed up with devs saying Windows was awful when they didn’t get how to use it. Things in that bucket included not remoting in to do syswork (use WMI/WinRM), not doing raw text munging unless you actually have to (COM from VBScript/PowerShell are your friends), adapting to the UAC model v. thesudomodel, etc. The Windows way can actually be very nice, but untraining habits is indeed hard.-
I don’t disagree with any of that (except maybe that I’m being harsh), but if you parse what I’m saying as “Windows is awful” then it’s because my indelicate tone has been read into instead of my words.
The point of the article is that those differences are superficial, and mean so very little to the mental model of use and implementation as to make no difference: IOCP is just threads and epoll, and epoll is just IOCP and fifos. Yes, IOCP is better, but I desperately want to see something new in how I use an operating system.
I’ve been doing things roughly the same way for nearly four decades, despite the fact that I’ve done Microsoft/IBM for a decade, Linux since Slackware 1.1 (Unix since tapes of SCO), Common Lisp (of all things) for a decade, and OSX for nearly that long. They’re all the same, and that point is painfully clear to anyone who has actually used these things at a high level: I edit files, I copy files, I run programs. Huzzah.
But: It’s also obvious to me who has gone into the bowels of these systems as well: I wrote winback which was for a long time the only tools for doing online Windows backups of standalone exchange servers and domain controllers; I’m the author of (perhaps) the fastest Linux webserver; I wrote ml a Linux emulator for OSX; I worked on ECL adding principally CL exceptions to streams and the Slime implementation. And so on.
So: I understand what you mean when you say Windows is not Linux, but I also understand what the author means when he says they’re the same.
-
-
I didn’t use the term “meaningfully different”, so I might be interpreting your question you too broadly.
When I used VMS, I never “made a backup” before I changed a file. That’s really quite powerful.
The Canon Cat had “pages” you would scroll through. Like other forth environments, if you named any of your blocks/documents it was so you could search [leap] for them, not because you had hierarchy.
I also think containers are very interesting. The encapsulation of the application seems to massively change the way we use them. Like the iOS example, they don’t seem to need “files” since the files live inside the container/app. This poses some risk for data portability. There are other problems.
I never used Oberon or WinRT enough to feel as comfortable commenting about them as I do about some of these other examples.
-
-
-
If it’s any motivation I would love to read this book.
Do you know of any books or posts I could read in the meantime? I’m very open to the idea that Windows is nice if you know which tools and mental models to use, but kind of by definition I’m not sure what to Google to find them :)
-
I’ve just been hesitant because I worked in management for two years after I started the book (meaning my information atrophied), and now I don’t work with Windows very much. So, unfortunately, I don’t immediately have a great suggestion for you. Yeah, you could read Windows Internals 6, which is what I did when I was working on the book, but that’s 2000+ pages, and most of it honestly isn’t relevant for a normal developer.
That said, if you’ve got specific questions, I’d love to hear them. Maybe there’s a tl;dr blog post hiding in them, where I could salvage some of my work without completing the entire book.
-
-
I, for one, would pay for your “Windows is not Linux” book. I’ve been developing for Windows for about 15 years, but I’m sure there are still things I could learn from such a book.
-
-
but users don’t use any of the VMSy goodness in Windows: To them it’s just another shitty UNIX clone, with everything being a file or a program (which is also a file). I think that’s the point.
Most users don’t know anything about UNIX and can’t use it. On the UI side, pre-NT Windows was a Mac knockoff mixed with MSDOS which was based on a DOS they got from a third party. Microsoft even developed software for Apple in that time. Microsoft’s own users had previously learned MSDOS menu and some commands. Then, they had a nifty UI like Apple’s running on MSDOS. Then, Microsoft worked with IBM to make a new OS/2 with its philosophy. Then, Microsoft acquired OpenVMS team, made new kernel, and a new GUI w/ wizard-based configuration of services vs command line, text, and pipes like in UNIX.
So, historically, internally, layperson-facing, and administration, Windows is a totally different thing than UNIX. Hence, the difficulty moving Windows users to UNIX when it’s a terminal OS with X Windows vs some Windows-style stuff like Gnome or KDE.
You’re also overstating the everything is a file by conflating OS’s that store programs or something in files vs those like UNIX or Plan 9 that use file metaphor for about everything. It’s a false equivalence: from what I remember, you don’t get your running processes in Windows by reading the filesystem since they don’t use that metaphor or API. It’s object based with API calls specific to different categories. Different philosophy.
-
Bitsavers has some internal emails from DEC at the time of David Cutler’s departure.
I have linked to a few of them.
David Cutler’s team at DECwest was working on Mica (an operating system) for PRISM (a RISC CPU architecture). PRISM was canceled in June of 1988. Cutler resigned in August of 1988 and 8 other DECwest alumni followed him at Microsoft.
-
-
-
-
-
It was edited by Simson Garfinkel, who co-wrote Building Cocoa Applications: a step-by-step guide. Which was sort of a “port” of Nextstep Programming Step One: object-oriented applications
Or, in other words, “yes” :)
-
Add me to the list curious about what they ended up using. The hoaxers behind UNIX admitted they’ve been coding in Pascal on Macs. Maybe it’s what the rest were using if not Common LISP on Macs.
-
-
-
Beat me to it. Author is full of it right when saying Windows is built on UNIX. Microsoft stealing, cloning, and improving OpenVMS into Windows NT is described here. This makes the Linux zealots’ parodies about a VMS desktop funnier given one destroyed Linux in desktop market. So, we have VMS and UNIX family trees going in parallel with the UNIX tree having more branches.
-
-
“we are forced to choose from: Windows, Apple, Other (which I shall refer to as “Linux” despite it technically being more specific). All of these are built around the same foundational concepts, those of Unix.”
Says it’s built on the foundational concepts of UNIX. It’s built on a combo of DOS, OS/2, OpenVMS, and Microsoft concepts they called the NT kernel. The only thing UNIX-like was the networking stack they got from Spider Systems. They’ve since rewritten their networking stack from what I heard.
-
Says it’s built on the foundational concepts of UNIX.
I don’t see any reason to disagree with that.
The only thing UNIX-like …
I don’t think that’s a helpful definition of “unix-like”.
It’s got files. Everything is a file. Windows might even be a better UNIX than Linux (since UNC)
Cutler might not have liked UNIX very much, but Windows NT ended up UNIX anyway because none of that VMS-goodness (Versions, types, streams, clusters) ended up in the hands of Users.
-
It’s got files. Everything is a file.
Windows is object-based. It does have files which are another object. The files come from MULTICS which UNIX also copied in some ways. Even the name was a play on it: UNICS. I think Titan invented the access permissions. The internal model with its subsystems were more like microkernel designs running OS emulators as processes. They did their own thing for most of the rest with the Win32 API and registry. Again, not quite how a UNIX programming guide teaches you to do things. They got clustering later, too, with them and Oracle using the distributed, lock approach from OpenVMS.
Windows and UNIX are very different in approach to architecture. They’re different in how developer is expected to build individual apps and compose them. It wasn’t even developed on UNIX: they used OS/2 workstations for that. There’s no reason to say Windows is ground in the UNIX philosophy. It’s a lie.
“Windows NT ended up UNIX anyway because none of that VMS-goodness (Versions, types, streams, clusters) ended up in the hands of Users.”
I don’t know what you’re saying here. Neither VMS nor Windows teams intended to do anything for UNIX users. They took their own path except for networking for obvious reasons. UNIX users actively resisted Microsoft tech, too. Especially BSD and Linux users that often hated them. They’d reflexively do the opposite of Microsoft except when making knockoffs of their key products like Office to get desktop users.
-
Windows is object-based.
Consider what methods of that “object” a program like Microsoft Word must be calling besides “ReadFile” and “WriteFile”.
That the kernel supports more methods is completely pointless. Users don’t interact with it. Programmers avoid it. Sysadmins don’t understand it and get it wrong.
I don’t know what you’re saying here.
That is clear, and yet you’re insisting I’m wrong.
-
Except, that’s completely wrong.
I just started Word and dumped a summary of its open handles by object type:
C:\WINDOWS\system32>handle -s -p WinWord.exe Nthandle v4.11 - Handle viewer Copyright (C) 1997-2017 Mark Russinovich Sysinternals - www.sysinternals.com Handle type summary: ALPC Port : 33 Desktop : 1 Directory : 3 DxgkSharedResource: 2 DxgkSharedSyncObject: 1 EtwRegistration : 324 Event : 431 File : 75 IoCompletion : 66 IoCompletionReserve: 1 IRTimer : 8 Key : 171 KeyedEvent : 24 Mutant : 32 Process : 2 Section : 67 Semaphore : 108 Thread : 138 Timer : 7 Token : 3 TpWorkerFactory : 4 WaitCompletionPacket: 36 WindowStation : 2 Total handles: 1539Each of these types is a distinct kernel object with its own characteristics and semantics. And yes, you do create and interact with them from user-space. Some of those will be abstracted by lower-level APIs, but many are directly created and managed by the application. You’ll note the number of open “files” is a very small minority of the total number of open handles.
Simple examples of non-file object types commonly manipulated from user-land include Mutants (CreateMutex) and Semaphores (CreateSemaphore). Perhaps the most prominent example is manipulating the Windows Registry; this entails opening “Key” objects, which per above are entirely distinct from regular files. See the MSDN Registry Functions reference.
-
None of these objects can exist on a disk; they cannot persist beyond shutdown, and do not have any representation beyond their instantaneous in-memory instance. When someone wants an “EtwRegistration” they’re creating it again and again.
Did you even read the article? Or are you trolling?
-
None of these objects can exist on a disk; they cannot persist beyond shutdown, and do not have any representation beyond their instantaneous in-memory instance. When someone wants an “EtwRegistration” they’re creating it again and again.
Key objects do typically exist on disk. Albeit, the underlying datastore for the Registry is a series of files, but you never directly manipulate those files. In the same sense you may ask for
C:\whatever.txt, you may ask forHKLM:\whatever. We need to somehow isolate the different persisted data streams, and that isolation mechanism is a file. That doesn’t mean you have to directly manipulate those files if the operating system provides higher-level abstractions. What exactly are you after?From the article:
But in Unix land, this is a taboo. Binary files are opaque, say the Unix ideologues. They are hard to read and write. Instead, we use Text Files, for it is surely the path of true righteousness we have taken.
The Windows Registry, which is a core part of the operating system, is completely counter to this. It’s a bunch of large binary files, precisely because Microsoft recognised storing all that configuration data in plain text files would be completely impractical. So you don’t open a text file and write to it, you open a Registry key, and store data in it using one of many predefined data types (
REG_DWORD, etc…).Did you even read the article? Or are you trolling?
It sounds like you’re not interested in a constructive and respectful dialogue. If you are, you should work on your approach.
-
What exactly are you after?
Just go read the article.
It’s about whether basing our entire interactions with a computer on a specific reduction of verbs (read and write) is really exploring what the operating system can do for us.
That is a very interesting subject to me.
Some idiot took party to the idea that Windows basically “built on Unix” then back-pedalled it to be about whether it was based on the same “foundational” concepts, then chooses to narrowly and uniquely interpret “foundational” in a very different way than the article.
Yes, windows has domains and registries and lots of directory services, but they all have the exact same “file” semantics.
But now you’re responding to this strange interpretation of “foundational” because you didn’t read the article either. Or you’re a troll. I’m not sure which yet.
Read the article. It’s not well written but it’s a very interesting idea.
Each of these types is a distinct kernel object with its own characteristics and semantics
Why do you bring this up in response to whether Windows is basically the same as Unix? Unix has lots of different kernel “types” all backed by “handles”. Some operations and semantics are shared by handles of different types, but some are distinct.
I don’t understand why you think this is important at all.
It sounds like you’re not interested in a constructive and respectful dialogue. If you are, you should work on your approach.
Do you often jump into the middle of a conversation with “Except, that’s completely wrong?”
Or are you only an asshole on the Internet?
-
-
-
-
-
-
-
The TCP/IP protocols were part of a UNIX. AT&T gave UNIX away for free. They spread together with early applications being built on UNIX. Anyone reusing the protocols or code will inherit some of what UNIX folks were doing. They were also the most mature networking stacks for that reason. It’s why re-using BSD stacks was popular among proprietary vendors. On top of the licensing.
Edit: Tried to Google you a source talking about this. I found one that mentions it.
-
-
-
-
-
-
-
-
Personally I think these small language are much more exciting than big oil tankers like Rust or Swift.
I’m not familiar with either of those languages, but any idea what the author means by this? I thought Rust has been picking up quite a bit recently.
-
I understood the author to be talking about the “size” of the language, not the degree of adoption.
I’m not sure that I personally agree that C is a small language, but many do belive that.
-
-
He is right though. C’s execution model may be conceptually simple but you may need to sweat the implementation details of it, depending on what you’re doing. This doesn’t make C bad, it just raises the bar.
-
I had that opinion before Rust, and I’m certainly not speaking on behalf of the Rust team, so in my understanding, the hat is very inappropriate.
(I’m also not making any claims about Rust’s size, in absolute terms nor relative to C)
-
Or you can just test his claim with numbers. A full, C semantics is huge compared to something like Oberon whose grammar fits on a page or two. Forth is simpler, too. Whereas, Ada and Rust are complicated as can be.
-
I agree that there are languages considerably smaller than C. In my view, there is a small and simple core to C that is unfortunately complicated by some gnarly details and feature creep. I’ve expressed a desire for a “better C” that does all we want from C without all the crap, and I sincerely believe we could make such a thing by taking C, stripping stuff and fixing some unfortunate design choices. The result should be the small and simple core I see in C.
When comparing the complexity of languages, I prefer to ignore syntax (focusing on that is kinda like bickering about style; yeah I have my own style too, and I generally prefer simpler syntax). I also prefer to ignore the standard library. What I would focus on is the language semantics as well as the burden they place on implementation. I would also weigh languages against the features they provide; otherwise we’re talking apples vs oranges where one language simply makes one thing impossible or you have to “invent” that thing outside the language spec. It may look simpler to only present a floating 64-bit point numeric type, but that only increases complexity when people actually need to deal with 64-bit integers and hardware registers.
That brings us to Oberon. Yes, the spec is short. I guess that’s mostly not because it has simple semantics, but because it lacks semantics. What is the range of integer types? Are they bignums, and if so, what happens you run out of memory trying to perform multiplication? Perhaps they have a fixed range. If so, what happens when you overflow? What happens if you divide by zero? And what happens when you dereference nil? No focking idea.
The “spec” is one for a toy language. That is why it is so short. How long would it grow if it were properly specified? Of course you could decide that everything the spec doesn’t cover is undefined and maybe results in program termination. That would make it impossible to write robust programs that can deal with implementation limitations in varying environments (unless you have perfect static analysis). See my point about apples vs oranges.
So the deeper question I have is: how small can you make a language with
- a spec that isn’t a toy spec
- not simply shifting complexity to the user
- enough of the same facilities we have in C so that we can interface with the hardware as well as write robust programs in the face of limited & changing system resources
Scheme, Oberon, PostScript, Brainfuck, etc. don’t really give us any data points in that direction.
-
So the deeper question I have is: how small can you make a language with
- a spec that isn’t a toy spec
- not simply shifting complexity to the user
- enough of the same facilities we have in C so that we can interface with the hardware as well as write robust programs in the face of limited & changing system resources
Scheme, Oberon, PostScript, Brainfuck, etc. don’t really give us any data points in that direction.
Good question. There are few languages with official standards (sorted by page count) that are also used in practice (well.. maybe not scheme ;>):
- Scheme r7rs - 88 pages - seems to be only language without useful standard library
- Ruby 1.8 - 341 pages
- Ada 95 - 582 pages
- Fortran 2008 - 621 pages - seems to be only language without useful standard library
- C11 - 701 pages
- EcmaScript - 885 pages
- Common Lisp - 1356 pages
- C++17 - 1623 pages
I know that page count is poor metric, but it looks like ~600 pages should be enough :)
-
Here are the page counts for a few other programming language standards:
- PL/I General purpose subset 443 pages
- Modula-2 800 pages - base - 707 pages, generics - 45 pages, objects - 48 pages
- Ada 2012 832 pages
- Eiffel 172 pages
- ISO Pascal 78 pages
- Jovial J73 168 pages
-
I know that page count is poor metric, but it looks like ~600 pages should be enough :)
Given that N1256 is 552 pages, yeah, without a doubt.. :-)
The language proper, if we cut it off starting at “future language directions” (then followed by standard library, appendices, index, etc.) is only some 170 pages. It’s not big, but I’m sure it could be made smaller.
-
I’ve expressed a desire for a “better C” that does all we want from C without all the crap, and I sincerely believe we could make such a thing by taking C, stripping stuff and fixing some unfortunate design choices. The result should be the small and simple core I see in C.
That might be worth you writing up with hypothetical design. I was exploring that space as part of bootstrapping for C compilers. My design idea actually started with x86 assembler trying to design a few, high-level operations that map over it which also work on RISC CPU’s. Expressions, 64-bit scalar type, 64-bit array type, variables, stack ops, heap ops, expressions, conditionals, goto, and Scheme-like macros. Everything else should be expressable in terms of the basics with the macros or compiler extensions. The common stuff gets a custom, optimized implementation to avoid macro overhead.
“ What I would focus on is the language semantics as well as the burden they place on implementation. “
Interesting you arrived at that since some others and I talking verification are convinced a language design should evolve with a formal spec for that reason. It could be as simple as Abstract, State Machines or as complex as Isabelle/HOL. The point is the feature is described precisely in terms of what it does and its interaction with other features. If one can’t describe that precisely, how the hell is a complicated program using those same features going to be easy to understand or predict? As an additional example, adding a “simple, local change” show unexpected interactions or state explosion once you run the model somehow. Maybe not so simple or local after all but it isn’t always evident if just talking in vague English about the language. I was going to prototype the concept with Oberon, too, since it’s so small and easy to understand.
“but because it lacks semantics.”
I didn’t think about that. You have a good point. Might be worth formalizing some of the details to see what happens. Might get messier as we formalize. Hmm.
“So the deeper question I have is: how small can you make a language with”
I think we have answers to some of that but they’re in pieces across projects. They haven’t been integrated into the view you’re looking for. You’ve definitely given me something to think about if I attempt a C-like design. :)
-
-
-
-
-
He is not claiming that memory safety in general is not an issue in C. What he is saying is that in his own projects he was able to limit or completely eliminate dynamic memory allocation:
In the 32 kloc of C code I’ve written since last August, there are only 13 calls to malloc overall, all in the sokol_gfx.h header, and 10 of those calls happen in the sokol-gfx initialization function
The entire 8-bit emulator code (chip headers, tests and examples, about 12 kloc) doesn’t have a single call to malloc or free.
That actually sounds like someone who understands that memory safety is very hard and important.
-
-
-
I’m not familiar with either of those languages, but any idea what the author means by this?
I’m also way more interested in Zig than I am in Rust.
What I think he’s saying is that the two “big” languages are overhyped and have gained disproportionate attention for what they offer, compared to some of the smaller projects that don’t hit HN/Lobsters headlines regularly.
Or maybe it’s a statement w.r.t. size and scope. I don’t know Swift well enough to say if it counts as big. But Rust looks like “Rubyists reinvented C++ and claim it to be a replacement for C.” I feel that people who prefer C are into things that small and simple. C++ is a behemoth. When your ideal replacement for C would also be small and simple, perhaps even more so than C itself, Rust starts to seem more and more like an oil tanker as it goes the C++ way.
-
I agree with your point on attention. I just wanted to say maybe we should get a bit more credit here:
“compared to some of the smaller projects that don’t hit HN/Lobsters headlines regularly.”
Maybe HN but Lobsters covers plenty oddball languages. Sometimes with good discussions, too. We had authors of them in it for a few. I’ve stayed digging them up to keep fresh ideas on the site.
So, we’re doing better here than most forums on that. :)
-
-
-
-
I always laugh when people come up with convoluted defenses for C and the effort that goes into that (even writing papers). Their attachment to this language has caused billions if not trillions worth of damages to society.
All of the defenses that I’ve seen, including this one, boil down to nonsense. Like others, the author calls for “improved C implementations”. Well, we have those already, and they’re called Rust, Swift, and, for the things C is not needed for, yes, even JavaScript is better than C (if you’re not doing systems-programming).
-
Their attachment to this language has caused billions if not trillions worth of damages to society.
Their attachment to a language with known but manageable defects has created trillions if not more in value for society. Don’t be absurd.
-
[citation needed] on the defects of memory unsafety being manageable. To a first approximation every large C/C++ codebase overfloweth with exploitable vulnerabilities, even after decades of attempting to resolve them (Windows, Linux, Firefox, Chrome, Edge, to take a few examples.)
-
Compared to the widely used large codebase in which language for which application that accepts and parses external data and yet has no exploitable vulnerabilities? BTW: http://cr.yp.to/qmail/guarantee.html
-
Your counter example is a smaller, low-featured, mail server written by a math and coding genius. I could cite Dean Karnazes doing ultramarathons on how far people can run. That doesn’t change that almost all runners would drop before 50 miles, esp before 300. Likewise with C code, citing the best of the secure coders doesn’t change what most will do or have done. I took author’s statement “to first approximation every” to mean “almost all” but not “every one.” It’s still true.
Whereas, Ada and Rust code have done a lot better on memory-safety even when non-experts are using them. Might be something to that.
-
-
That’s cheating saying “non-c” and “widely used.” Most of the no-error parsing systems I’ve seen use a formal grammar with autogeneration. They usually extract to Ocaml. Some also generate C just to plug into the ecosystem since it’s a C/C++-based ecosystem. It’s incidental in those cases: could be any language since the real programming is in the grammar and generator. An example of that is the parser in Mongrel server which was doing a solid job when I was following it. I’m not sure if they found vulnerabilities in it later.
-
-
-
-
There was an integer overflow reported in qmail in 2005. Bernstein does not consider this a vulnerability.
-
-
-
That’s not what I meant by attachment. Their interest in C certainly created much value.
-
-
-
Tony Hoare, null’s creator, regrets its invention and says that just inserting the one idea has cost billions. He mentions it in talks. It’s interesting to think that language creators even think of the mistakes they’ve made have caused billions in damages.
“I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn’t resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.
If the billion dollar mistake was the null pointer, the C
getsfunction is a multi-billion dollar mistake that created the opportunity for malware and viruses to thrive.-
He’s deluded. You want a billion dollar mistake: try CSP/Occam plus Hoare Logic. Null is a necessary byproduct of implementing total functions that approximate partial ones. See, for example, McCarthy in 1958 defining a LISP search function with a null return on failure. http://www.softwarepreservation.org/projects/LISP/MIT/AIM-001.pdf
-
“ try CSP/Occam plus Hoare Logic”
I think you meant formal verification, which is arguable. They could’ve wasted a hundred million easily on the useless stuff. Two out of three are bad examples, though.
Spin has had a ton of industrial success easily knocking out problems in protocols and hardware that are hard to find via other methods. With hardware, the defects could’ve caused recalls like the Pentium bug. Likewise, Hoare-style logic has been doing its job in Design-by-Contract which knocks time off debugging and maintenance phases. The most expensive. If anything, not using tech like this can add up to a billion dollar mistake over time.
Occam looks like it was a large waste of money, esp in the Transputer.
-
-
-
-
-
“especially when you consider what wouldn’t have been built without C.”
I just countered that. The language didn’t have to be built the way it was or persist that way. We could be building new stuff in a C-compatible language with many benefits of HLL’s like Smalltalk, LISP, Ada, or Rust with the legacy C getting gradually rewritten over time. If that started in the 90’s, we could have equivalent of a LISP machine for C code, OS, and browser by now.
-
It didn’t have to, but it was, and it was then used to create tremendous value. Although I concur with the numerous shortcomings of C, and it’s past time to move on, I also prefer the concrete over the hypothetical.
The world is a messy place, and what actually happens is more interesting (and more realistic, obviously) than what people think could have happened. There are plenty of examples of this inside and outside of engineering.
-
The major problem I see with this “concrete” winners-take-all mindset is that it encourages whig history which can’t distinguish the merely victorious from the inevitable. In order to learn from the past, we need to understand what alternatives were present before we can hope to discern what may have caused some to succeed and others to fail.
-
Imagine if someone created Car2 which crashed 10% of the time that Car did, but Car just happened to win. Sure, Car created tremendous value. Do you really think people you’re arguing with think that most systems software, which is written in C, is not extremely valuable?
It would be valuable even if C was twice as bad. Because no one is arguing about absolute value, that’s a silly thing to impute. This is about opportunity cost.
Now we can debate whether this opportunity cost is an issue. Whether C is really comparatively bad. But that’s a different discussion, one where it doesn’t matter that C created value absolutely.
-
-
-
-
-
Billions of lines of COBOL run mission-critical services of the top 500 companies in America. Better to research the causes of this than laughing it off. Are you ready to give up C for COBOL on mainframes or you think both of them’s popularity were caused by historical events/contexts with inertia taking over? Im in latter camp.
-
Are you ready to give up C for COBOL on mainframes or you think both of them’s popularity were caused by historical events/contexts with inertia taking over? Im in latter camp.
Researching the causes of something doesn’t imply taking a stance on it, if anything, taking a stance on something should hopefully imply you’ve researched it. Even with your comment I still don’t see how laughing off a fact is better than researching its causes.
You might be interested in laughing about all the cobol still in use, or in research that looks into the causes of that. I’m in the latter camp.
-
-
Cobol has some interesting features(!) that make it very “safe”. Referring to the 85 standard:
X. No runtime stack, no stack overflow vulnerabilities Y. No dynamic memory allocation, impossible to consume heap Z. All memory statically allocated (see Y); no buffer overflows -
We should use COBOL with contracts for transactions on the blockchains. The reasons are:
X. It’s already got compilers big businesses are willing to bet their future on.
Y. It supports decimal math instead of floating point. No real-world to fake, computer-math conversions needed.
Z. It’s been used in transaction-processing systems that have run for decades with no major downtime or financial losses disclosed to investors.
λ. It can be mathematically verified by some people who understand the letter on the left.
You can laugh. You’d still be missing out on a potentially $25+ million opportunity for IBM. Your call.
-
-
You’re right! I’ll pitch IBM’s senior executives on it the first chance I get. I’ll even put on a $600 suit so they know I have more business acumen than most coin pitchers. I’ll use phrases like vertical integration of the coin stack. Haha.
-
-
-
-
That makes sense. I did do the C research. Ill be posting about that in a reply later tonight.
-
Ill be posting about that in a reply later tonight.
Good god man, get a blog already.
Like, seriously, do we need to pass a hat around or something? :P
-
Haha. Someone actually built me a prototype a while back. Makes me feel guilty that I dont have one instead of the usual lazy or overloaded.
-
-
That’s cool. Setting one up isn’t the hard part. The hard part is doing a presentable design, organizing the complex activities I do, moving my write-ups into it adding metadata, and so on. I’m still not sure how much I should worry about the design. One’s site can be considered a marketing tool for people that might offer jobs and such. I’d go into more detail but you’d tell me “that might be a better fit for Barnacles.” :P
-
-
This.
Hell, Charles Bloom’s blog is basically an append-only textfile.
-
-
-
Making me feel guilty again. Nah, I’ll build it myself likely on a VPS.
And damn time has been flying. Doesnt feel like several months have passed on my end.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Is there a reason why you believe that other languages don’t translate to assembly well?
It’s true those other languages are harder to implement, but it seems to be a moot point to me when compilers for them already exist.
-
Some users of C need an assembly-level understanding of what their code does. With most other languages that isn’t really achievable. It is also increasingly less possible with modern C compilers, and said users aren’t very happy about it (see various rants by Torvalds about braindamaged compilers etc.)
-
“Some users of C need an assembly-level understanding of what their code does.”
Which C doesnt give them due to compiler differences and effects of optimization. Aside from spotting errors, it’s why folks in safety- critical are required to check the assembly against the code. The C language is certainly closer to assembly behavior but doesnt by itself gives assembly-level understanding.
-
-
-
-
-
-
“but the best operating systems are still written in it.”
That’s an incidental part of history, though. People who are writing, say, a new x86 OS with a language balancing safety, maintenance, performance, and so on might not choose C. At least three chose Rust, one Ada, one SPARK, several Java, several C#, one LISP, one Haskell, one Go, and many C++. Plenty of choices being explored including languages C coders might say arent good for OS’s.
Additionally, many choosing C or C++ say it’s for existing tooling, tutorials, talent, or libraries. Those are also incidental to its history rather than advantages of its language design. Definitely worthwhile reasons to choose a language for a project but they shift the language argument itself implying they had better things in mind that werent usable yet for that project.
-
-
Oh OK. My bad. That’s a reasonable position.
-
-
-
-
-
-
We’ve dabbled with SMS, but some of these records are just too large and complex for SMS.
What would be the typical size for these records? Ordinary sms has 140 bytes, but you could use some binary serialization and concatenated SMS (pdu mode) to get more.
-
We’re generally under 50kb, but we have contact trace forms that have up to 200 fields in them, some variable text length. I did a bit of looking into SMS but part of the problem is getting sims and an SMS gateway in place in country. At one point we were in talks with a local telecom for almost 3-4 months to get a gateway set up before they just stopped responding to our emails and they wouldn’t give us any specs or docs on how their gateways work or even what gateway they were using. This is very typical.
-
-
-
Why not use forward declarations without public definitions to create opaque types? For example, in the public header file:
typedef struct handle handle_t;Then, only in the code which needs to access the concrete type:
struct handle { ... };Then the compiler can correctly allow a
handle_t *only where the function signature wants one.-
Incomplete types as abstractions has a complete example of doing this.
-
-
I really don’t understand these manifestos. This feels like an extension of look-at-me culture with little substance but a ton of platitudes.
We can add the most value as professionals by drawing on the diversity of our identities, backgrounds, experiences, and perspectives. Homogeneity is an antipattern.
This already happens. I’ve worked at a company that was mostly white males, but none of us had very similar backgrounds. Some would argue our skin color and genitals would make us homogenous, but it’s not true. We all had vastly different experiences in getting to where we were (and there were only 2 of us in software out of like 15 people).
How do you hire for a heterogenous company anyway? If you base it on superficial things like skin color, you’re racist. If you base it on gender, you’re sexist. I’m talking about people on both sides here - you can’t use those as qualifiers for how good someone is at their job. You also can’t ask their personal history and sexuality. So what are you trying to do here?
It feels like another one of those “too many white/males in the industry, we need to diversify” without realizing that not all whites/males are the same. Surely if I said all black males are the same, someone would question it.
We can be successful while leading rich, full lives. Our success and value is not dependent on exerting all of our energy on contributing to software.
I’ve never worked at a place that acted differently than this. I’m not saying those types of places don’t exist, because I’m sure they do. I’ve worked at a fair number of companies, though, big and small, and no one has acted like this.
We have the obligation to use our positions of privilege, however tenuous, to improve the lives of others.
What does this even mean?
We must make room for people who are not like us to enter our field and succeed there. This means not only inviting them in, but making sure that they are supported and empowered.
No one is like me, and I welcome anyone to try out what I do if they’re interested. Beyond giving them feedback and code reviews, I don’t know what else people want. I didn’t even get that when I was coming up, but I had the desire to keep learning and pushing myself regardless of who I worked with. I feel like this may border on hand-holding.
We have an ethical responsibility to refuse to work on software that will negatively impact the well-being of other people.
This is a total judgment call on what negatively impacts the well-being of other people. If I worked for the cattle industry, I could be deemed as hurting the well-being of other people (deforestation, hormones, etc). I work for the credit card industry right now, which absolutely hurts the well-being of some people. It helps others. I don’t feel like I’m in a predatory company, though. I wouldn’t even bother with a place that I felt was that way.
We acknowledge the value of non-technical contributors as equal to the value of technical contributors.
Hmm, sort of. It depends on the role. There are definitely some roles that do NOT match the value of technical contributions, but there are others that are vastly more important. That goes for any role, though. I’m not sure what this even means.
We understand that working in our field is a privilege, not a right. The negative impact of toxic people in the workplace or the larger community is not offset by their technical contributions.
This would apply to any career path. You don’t really have a right to be in any field. So ..?
We are devoted to practicing compassion and not contempt. We refuse to belittle other people because of their choices of tools, techniques, or languages.
You probably write PHP, don’t you? (I’m kidding!!)
The field of software development embraces technical change, and is made better by also accepting social change.
I don’t see the connection between software development and social change. Gay marriage being legalized does literally nothing to my list. Nor does it change anything in my credit score application for work. I mean, I guess if you’re saying you’re in the wedding cake biz and you now get more customers, sure, you’re going to embrace that. This is a huge stretch, though.
–
I’m done ranting, I guess. I’m obviously not going to sign this and I would implore anyone thinking of doing so to ask themselves why they are. If you’re thinking of signing it because something speaks to you, I would ask you to “be the change you want to see” despite how corny that is. Signing this doesn’t mean anything, acting in a way you feel best expresses your values does though!
I try to stick to the golden rule, despite how jerky some people can be (including myself). We’re all going to be dead soon, so make the best of it that you can.
-
I do want to pick on one thing here:
I don’t see the connection between software development and social change.
As a non-male, non-female person, technologists pretending I can’t exist is a right pain in the ass. Having to choose between “male” and “female” without any explanation of the effects of the decision and no option to not choose on many sites is quite aggravating.
Similar issues arise with less contentious personal data: for instance, several applications I’ve used assume I have only gone to one college, while in reality I have separate credits from a local community college and my 4-year degree program.
Technologists literally encode social norms into software, and it is our responsibility not to shut people out of those systems artificially.
-
-
-
-
That’s a neat presentation, but for those of us who haven’t raised 1.5B in funding there are usually more pressing engineering concerns.
-
Pinterest is quite a different environment than my employer.
Here are a few of the differences:
- Most applications are developed solely for internal use.
- Turnover is low, reorganizations are rare, and there isn’t a lot of chaos.
- There are about a dozen extremely busy developers.
- Software systems are owned by an individual or team.
- The division that funds the development of an application decides on the scope of a project.
We may not have any applications that ask the user for a gender identity.
If there are any, then the options were determined by HR and the division that funded the application.-
We may not have any applications that ask the user for a gender identity.
This is, I think, a key point. There are not many reasons to ask for a gender identity. From the top of my head:
- You’re doing medicine and you mean sex. In that case, ask about sex, but still don’t leave out intersex people or you will have a bad time.
- It will be used to determine preference for partners. This is perhaps the most complex case as it does absolutely require canonicalization.
- It will be passed to a third-party system that has no understanding of nonbinary gender. In this case, that third-party system sucks, but it’s not morally an issue for you (since the decision has been made elsewhere). Putting pressure on the other system to change would be good but may not be possible.
- It is meant to be displayed somewhere, like a profile. In this case, probably an arbitrary string is fine, perhaps with suggested options.
- You want to use it to target ads. Don’t. That’s going to lead you down a rabbit hole of sexism and inaccurate targetting. See, for example, several trans women I know who started getting ads for women-focused products which they wanted long before they came out to anyone, because Google uses behavior, not stated gender, to target ads.
- You want demographic info. In that case, the whole point is to represent reality as closely as possible, so it seems hard to justify not at least adding an “Other” option.
This is a very small subset of apps (for example: Pinterest, why?)
-
-
-
-
-
-
technologists pretending I can’t exist is a right pain in the ass
That’s neither a fair nor charitable reading of what’s almost certainly going on. It’s vastly, vastly more likely that the developers were making tradeoffs on time and implementation complexity, and went with the choice that frankly works in several sigmas of the population. Engineering requires limiting problem domains, and while I’m sorry you fell on the wrong side of that line, it’s a valid one.
EDIT:
I have seen that particular invocation many times, among friends, family, and others–if somebody immediately makes everything a life or death matter it becomes very difficult to have productive discussions (as one would expect!).
-
That’s neither a fair nor charitable reading of what’s almost certainly going on. It’s vastly, vastly more likely that the developers were making tradeoffs on time and implementation complexity, and went with the choice that frankly works in several sigmas of the population.
I think that’s also uncharitable, in that it assumes there’s intent. I think the most likely case is ignorance. Realizing that having “male” and “female” is insufficient is a matter of life-experiences: if you’re not nonbinary and don’t know anybody who isn’t nonbinary, then you’re not going to realize you should be using nonbinary options on your site.
This isn’t only a problem with inclusion and social norms. See what can you put in a refrigerator for an example of how lacking life-experience can lead to you missing edge cases in your systems.
Engineering requires limiting problem domains, and while I’m sorry you fell on the wrong side of that line, it’s a valid one.
This gets into “what’s efficient” vs “what’s just”. Just because a tradeoff is valid engineering-wise doesn’t mean we have to accept it socially… nor that the status quo is what we should accept, socially. Remember, the Americans with Disabilities Act is less than 30 years old.
-
… it’s a valid one.
In other words, fuck me for not being willing to conform in a way that makes another software developer’s job very slightly harder. This is literally exactly the attitude I’m highlighting as harmful. There are many more assumptions we encode into our software - for instance, basing a great deal on the ability to see, or even to see in color - that are harmful to subsets of our users.
-
fuck me for not being willing to conform in a way that makes another software developer’s job very slightly harder.
There are a dozen possible reasons that that field is implemented as it exists, and I will bet you lunch and a beer that none of them are “Let’s find another way of passive-aggressively fucking with trans or non-binary folks”. Many of those reasons are probably more complex, either due to technological or organizational reasons, than you are willing to give them credit for.
-
Agreed, 100%! That’s exactly how systemic bias works: few people consciously choose to screw over the disadvantaged group; instead, problems come from simple ignorance or willful neglect of that minority’s needs. That allows them (and, apparently, you) to justify ignoring complaints from that group without making the effort to improve the situation…
-
Yes, I suppose there is a systematic bias in engineering towards getting the most benefit from the least effort.
Also, please don’t assume anything about the software I write or have written or the issues I ran into trying to support edgecases. It makes you come off as shrill and petty.
I don’t assume your gender, please don’t assume my history.
-
there is a systematic bias in engineering towards getting the most benefit from the least effort
The most benefit for whom? This is precisely the question the manifesto asks.
please don’t assume anything about the software I write or have written or the issues I ran into trying to support edgecases
That wasn’t an assumption about anything; it was a comment on how you did the thing I described in the post I was replying to.
I don’t assume your gender
Do you… really think trans people say that? Like ever? Because I know a lot of trans people and I have never heard anyone use the phrase “assume my gender” unironically.
-
The most benefit for whom? This is precisely the question the manifesto asks.
Let’s say, for the sake of argument, that it takes X man-hours to implement feature F that benefits A users. Let’s assume it takes (X+Y) man-hours to implement feature F’ that benefits (A+B) users. We assume that F’ is a strict superset of F functionality, and that B is strictly positive. We assume that the additional B users pay us the same amount as the A users.
We define the efficiency E as the number of users benefiting from some feature divided by the time it takes to implement that feature.
So, we get:
E(F) = A / X E(F') = (A+B) / (X+Y)Thus, it’s pretty obvious that the marginal benefit, even if non-zero for some users, can be dwarfed by costs if the implementation time is too great.
To hammer this home with a concrete example:
You have 10 users with feature A that took 1 man hour. You get an additional 2 users for implementing feature A’ instead, which takes 2 man hours. Your efficiency has dropped from 10 to 6 because you went after those 2 users.
This is neither biased (except towards efficient allocation of resources) nor particularly complicated a calculation to make.
Genuine question: have you not ever had to deal with sacrificing features because of limited engineering resources and time constraints, even features you knew were important to somebody?
-
This is neither biased (except towards efficient allocation of resources) nor particularly complicated a calculation to make.
This is not a small bias however. It’s a huge cultural bias with possibly surprising effects.
When I was a young and inexpert web developer (or in current parlance, a full stack developer) I was asked to add online payment to a web site used by turists from all around the world.
I studied the alternatives: back then few italian banks provided viable online payment solutions, and Paypal was still not widely known in Italy but I studied it too. After a 2 or 3 days, I talked with my boss about the solutions I had in mind, their pros and cons and the effort required.
My boss told me something like “No, I want you to add an input box to the registration form so that turists can put their credit card in, and record it in a new column of the user table. It’s at most one day of work and you are late.”
I remember I said: “Sorry, but I will not do that”.
The boss was pretty explicit about the fact that I cannot say “no” (it was a small italian ISP and we were just 3 web developers, most people there were system administrators). Being young and weird, this cristalized my position: I said “I think that, if that’s the only option, you should tell the customer they cannot have the feature, as I will not do that.”.
The boss was pretty angry, he escalated to the company owner that again explained me that I were not professional, that I should always do what they ask since they are accountable, that I was reducing the whole company efficiency just for phylosophy and that this would have had implications in my career and so on… but my position did not change.
After wasting half a day in three about this, they assigned the task to another developer that did it pretty rapidly.
One years later the website was hacked by a Brazilian crew that stolen all the data, including the credit cards. They were kind enough to thanks everyone in the home page.
The company got sued and settled to pay the customer around 4.000.000 euros of demage.
Trust me, being right that time was not funny. Not even when everybody understood I was right.
But it teached me a lesson: “efficient allocation of resources” is often an illusion.
-
Genuine question: have you not ever had to deal with sacrificing features because of limited engineering resources and time constraints, even features you knew were important to somebody?
Yes, but I take your point.
Your analysis is spot on, of course, as far as it goes. However, it only goes so far. We are forced to ask the question: does string gender really take twice as much effort to implement as bool gender? I seriously doubt it.
-
I’ve had this fight with several product managers. For those interested in practical ways to improve the situation, may I recommend digging into “why do you want this info” and incorporating that into the question.
For instance, one product would refer users to gender specific services; we could reword the question to “are you interested in services targeted to men/women/both/neither”.
In another case it was because they wanted more money for ad placements; we pretty much had to stick with binary gender as that’s what the advertisers paid for.
-
-
I feel like these discussions (hell, most social justice discussions) are impossible to hold in the abstract; it devolves into people talking past one another. Examples are the only way I’ve found to get past that.
-
@danielrheath and @dbremner both bring up most of the examples I would’ve gone with. Further annoying reasons exist like:
- Existing formulas require a binary
- Legacy systems don’t support it
- No, really, legacy systems
- Doing it correctly can be hard
If devs have to interop with any of the above, complexity of implementation gets a good deal more complicated.
-
Do note here the difference between gender and sex. If you’re doing medical info, you need to ask about sex, not gender.
Also note that formulas like those given for estimating BMR break down for people who don’t physically conform to ideas about what is “male” and “female”; intersex people, people on HRT, et cetera generally require personal contact with physicians or simply cannot be evaluated on those criteria. While this is not the responsibility of the engineer, neither is it a good argument for restricting the available options in the general case.
Regarding legacy systems, it’s very easy to canonicalize; the examples you give either ask about sex, not gender, or have a third option.
Fundamentally, you keep making points about how it’s difficult or annoying to implement this, but not doing so comes at a cost as well: such a system is incapable of representing reality in a fairly major way. That’s a valid choice to make, but dismissing the population of people whose bodies or minds you are unable to represent as non-significant seems shortsighted and somewhat foolish.
-
If you’re doing medical info, you need to ask about sex, not gender.
Go read the spec. It’s explicitly called “administrative gender”. I know that choice of words to be inaccurate, you know that to be inaccurate–but the important thing is that the legacy system in healthcare (and HL7/FHIR is a pretty big deal) doesn’t care. And the third option the standard gives is for intersex folks–so, is incorrect if used when that is not physically correct.
seems shortsighted and somewhat foolish.
Again, the recurring point is “Hey, yeah, the developer probably sees that there is a better answer, but fixing it when they run up against systems that are already doing it wrong is too expensive.”
Your original assertion way back was that the developers were doing this all out of malice or spite, and the thing all of us keep trying to tell you–because as devs we need to have some basic respect and charity towards each other until proven otherwise–is that there are many reasons for this thing to happen.
-
Go read the spec. It’s explicitly called “administrative gender”. I know that choice of words to be inaccurate, you know that to be inaccurate–but the important thing is that the legacy system in healthcare (and HL7/FHIR is a pretty big deal) doesn’t care.
Right, but “administrative gender” is a technical term meaning, effectively, “sex”, so we’re in agreement about how that should be represented. If the term must be used on, e.g., a patient registration form, explanatory text should be present (and usually is, in my experience).
Your original assertion way back was that the developers were doing this all out of malice or spite
That was very expressly not my assertion, which is why I brought up systemic bias. The fact that there are many reasons for it to happen does not lessen the negative impact and does not absolve the developer of all responsibility.
I absolutely agree that very few developers would implement a binary gender system out of explicit anti-transgender sentiment; rather, my assertion is that most developers care more about X than about accurately representing the gender and/or sex of people who don’t fit a binary model. That in and of itself is a moral decision, and one of the main points of the “manifesto” we are discussing is that moral decisions like that should not be dismissed, but should be examined as part of a consideration of engineering tradeoffs, and that whoever makes that decision needs to be morally responsible for it.
-
-
-
-
For those interested in practical ways to improve the situation, may I recommend digging into “why do you want this info” and incorporating that into the question.
I like this approach a lot, despite being skeptical about the manifesto.
It’s basically an application of domain driven design to a business process.
However, as a DDD practicioner myself, I must warn you that you are going to face a lot more complexity with this approach. Usually it’s worth the effort, but not every single time.
In DDD term you are asking a question related to a specific context (let’s call it “services of interest”), but you don’t know anything about the gender of the user. Another context might be “products of interests” (where products are physical goods, for example). Now, you have more precise informations about the users, the users knows what you know about them, but you need much more complexity to handle such informations. A lot more complexity if that informations are descriptive strings as @dbremner properly said.
Also, you might have Male and Female as non mutually exclusive checkboxes.
But there might be no point to do that if you offer no products or services targetting non-binary people.
-
-
It could just be me but using a string instead of a binary gender seems like a lot more than twice as much work.
I’m assuming that this is for a public-facing web application.
Here are some of the questions that I would ask if I were tasked with implementing this.
- Do we allow the user to enter a string or pick from a list?
- How many characters is the string permitted to be?
- What characters are valid?
- How is the gender stored in the database, one string per user or is it normalized?
- How does a long string affect our UI on popular browsers and mobile devices?
- Is the user input checked against a list of forbidden words?
- If we use a list of forbidden words, what is the process for adding new words to the list?
- Does adding an entry to the list of forbidden words affect existing profiles?
- Do we trim leading and trailing spaces or tabs?
- Is the gender string stored in a canonical format for reporting or analytics?
- If the user enters a string that we reject, how does the UI indicate this?
- What additional changes to the profile page are required?
All of these questions can be answered but I can see why Facebook uses a fixed list of strings.
-
-
-
-
-
-
-
In other words, fuck me for not being willing to conform in a way that makes another software developer’s job very slightly harder.
Or fuck every software developer in the world for not being willing to expend time and money changing their software to reflect things you believe in that they don’t. Take any widely-held belief with a lot of evidence behind it or just momentum that brings in profit as the businesses do by default. Take any individual or small group dissenting saying it should change to do arbitrary things they want. All the software should then be rewritten to do what the small group wants. If majority of users or customers disagree, it should still be rewritten to do what the small group wants saying fuck you to the majority. That’s basically your requirement.
That doesn’t seem fair or logical either. It could get pretty expensive and crazy unusable really quickly depending on what all the demands are from various groups. Instead, they should act on a combination of what the most objective evidence says and what’s good for them since it’s their work. In your case, that might work to your favor since the biological evidence shows significant exceptions to two genders with one type even being both. Some are ambiguous. So, an Other field with something user-supplied could work. I’m sure it would be abused but people could’ve been writing bullshit into the forms regardless. Folks can argue about the accuracy of whatever was on the form in email, court, or whatever if it matters enough to all involved. Companies will probably just take what was entered.
-
All the software should then be rewritten to do what the small group wants. If majority of users or customers disagree, it should still be rewritten to do what the small group wants saying fuck you to the majority.
But I don’t think the majority would actually be unhappy with this change. Your example is more like asking every website to change to a baby blue and pink color scheme - it would negatively impact the vast majority of users. This negatively impacts bigots (a minority) and perhaps slightly affects the bottom line (although really, how difficult/expensive is it to switch from a boolean/enum to a string in the database for gender?), and is neutral or beneficial for everyone else.
-
“But I don’t think the majority would actually be unhappy with this change.” “This negatively impacts bigots “
Anyone that disagrees with your gender claim is a bigot. Ok. So, let’s start there. Rather than a tiny change with no impact, this is a change that makes a specific political statement which will require support of the company/organization, them to defend the choice if it becomes a media event, and for their customers to be OK with it.
The political statement is highly contentious with the status quo being two genders. That model was created and sustained over tens of thousands of years because it applied to almost everyone in existence across billions of individuals. That makes it a sensible default where new labels will be used for rare exceptions. Many customers, due to their political beliefs, oppose changing that model. You’re saying the developers should, by default, oppose that model in their online forms or they’re bigots. I say they’re doing what works 99% of the time that might also reflect their beliefs and/or those they perceive of many customers. They might even fear dropping a political landmine into a signup process that’s supposed to create a positive experience with minimal friction. As in, the sign-ups are part of sales pipeline.
Like I said, I’d consider doing it in a carefully-worded way myself. I’m not surprised others don’t since they’re probably males/females who mostly know males/females whose customers mostly know or are males/females dropping male/female on a forum. I’m also not surprised they’d think carefully about the consequences of changing it. That makes sense, too. I’m glad people like in that one link are working on and deploying alternatives since that happening without any backlash could inspire more companies to take the chance on a change.
-
Anyone that disagrees with your gender claim is a bigot. […] You’re saying the developers should, by default, oppose that model in their online forms or they’re bigots.
No, that’s not what I said or meant. What I said was that people who disagree but aren’t enraged the moment they see evidence of a trans person would be neutrally, not negatively, impacted, meaning the changes aren’t a problem for them (like a stupid and hard to read color scheme would be).
That model was created and sustained over tens of thousands of years because it applied to almost everyone in existence across billions of individuals.
This is incorrect, and is the basis of a lot of incorrect beliefs about trans people in general. The Wikipedia article on third genders is a good primer, but there are many resources online and in books you might be interested in. The TL;DR is that tons of non-European cultures had more than two genders: Inca quariwarmi, Native American Two-Spirits, ancient Israeli androgynos and tumtum, and more. A major part of colonization (and Christianization) in the 18th and 19th centuries was the erasure of those non-European (and, as viewed by the Church, sinful) gender identities.
In addition, about 1.7% of people are born intersex, meaning they can’t be easily classified as “biologically male” or “biologically female” [1]. Many of these people are operated on at birth to make them more easily classifiable, but that almost one in fifty people are “born nonbinary” (and presumably this is not a modern creation, since it is biological in origin) should perhaps give you some perspective on the prevalence of NB identities.
You then go on to explain a lot of reasons why people don’t want to do this. The point that I think a lot of people miss is that I get that. I understand. So does the author of this manifesto, and her response is, “I will not be one of those people.” That is, in many ways, a major point of the document.
1: Fausto-Sterling, Anne (2000). Sexing the Body: Gender Politics and the Construction of Sexuality. Basic Books. p. 53. ISBN 978-0-465-07714-4.
-
-
-
-
-
-
-
-
Personal - ILTA is a volunteer organization for legal IT staff. Most law firms use similar technology stacks and we face a lot of the same issues. Last week an ILTA member at another firm asked about adding an autocorrection for an attorney’s name to all of their software systems. I have used the Windows Spell Checking API before so I wrote a command line program for adding autocorrections to the Windows dictionary. I also cleaned up my Github profile by deleting obsolete forks.
Work - I spent most of this week on user support requests.
-
Last week
- Fiddled with different ways of attaching to processes and viewing their states.
- Some other technical stuff that went well
This was for the low level debugger I’m trying to make.
So, from what I’ve read and seen, tools that attach and inspect other process tend to just use gdb under the hood. I was hoping for a more minimal debugger to read and copy.
lldbalmost does what I need because of its existing external Python interface but documentation for writing a stand-alone tool (started from outside the debugger rather than inside) is scattered. I haven’t managed to make it single step.Using raw
ptraceand trying to read the right memory locations seems difficult because of things like address randomization. And getting more information involves working with even more memory mapping and other conventions.I wish all these conventions were written in some machine readable language agnostic way so I don’t have to human-read each one and try to implement it. Right now this is all implicit in the source code of something like
gdb. This is a lot of extra complexity which has nothing to do with what I’m actually trying to accomplish.The raw ptrace approach would also likely only work for Linux. And possibly strong tied to C or assembly.
The problem with the latter is that eventually I will want to do this to interpreters written in C or even interpreters written in interpreters written in C. Seems like even more incidental complexity in that way.
An alternative is to log everything and have a much fancier log viewer after the fact. This way the debugged program only need to emit the right things to a file or stdout. But this loses the possibility for any interactivity.
Plus, all of this would only be worth it if I can get some state visualization customizable to that specific program (because usually it will be an interpreter).
Other questions: How to avoid duplicating the work when performing operations from “inside the program” and from “outside” through the eventual debugger?
Other ideas: Try to do this with a simpler toy language/system to get an idea of how well using such a workflow would work in the first place.
Some references
- https://eklitzke.org/ptrace
- https://backtrace.io/blog/backtrace/debugger-internals/
- https://blog.0x972.info/?d=2014/11/13/10/40/50-how-does-a-debugger-work
This week
- Well, now that I have a better idea of how deep this rabbit hole is, I need to decide what to do. Deciding is much harder than programming…
- Or maybe I should do one of the other thousand things I want to and have this bit of indecision linger some more.
-
I wrote a very simple PoC debugger in Rust if you are interested in the very basics: https://github.com/levex/debugger-talk
It uses
ptrace(2)under the hood, as you would expect.-
Thanks! I’ve a had a look at your slide and skimmed some of your code (don’t have Rust installed or running would be the first thing I’d do).
I see that you’re setting breakpoints by address. How do you figure out the address at which you want to set a breakpoint though?
How long did it take to make this? And can you comment on how hard it would be to continue from this point on? For example reading C variables and arrays? Or getting line numbers from the call stack?
-
Hey, sorry for the late reply!
In the talk I was setting breakpoints by address indeed. This is because the talked focused on the lower-level parts To translate line numbers into addresses and vice-versa you need access to the “debug information”. This is usually stored in the executable (as decribed by the DWARF file format). There are libraries that can help you with this (just as the disassembly is done by an excellent library instead of my own code).
This project took about a week of preparation and work. I was familiar with the underlying concepts, however Rust and its ecosystem was a new frontier for me.
Reading C variables is already done :-), reading arrays is just a matter of a new command and reading variables sequentially.
-
Thanks for coming back to answer! Thanks to examples from yourself and others I did get some stuff working (at least on the examples I tried) like breakpoint setting/clearing, variable read/write and simple function calls.
Some things from the standards/formats are still unclear, like why I only need to add the start of the memory region extracted from /proc/pid/maps if its not 0x400000.
This project took about a week of preparation and work. I was familiar with the underlying concepts, however Rust and its ecosystem was a new frontier for me.
A week doesn’t sound too bad. Unfortunately, I’m in the opposite situation using a familiar system to do something unfamiliar.
-
-
-
For C/assembly (and I’ll assume a modern Unix system) you’ll need to read up on ELF (object and executable formats) and DWARF (debugging records in an ELF file) that contain all that information. You might also want to look into the GDB remote serial protocol (I know it exists, but I haven’t looked much into it).
-
Well, I got some addresses out of
nm ./name-of-executablebut can’t peek at those directly. Probably need an offset of some sort?There’s also
dwarfdumpI haven’t tried yet. I’ll worry about how to get this info from inside my tracer a bit later.Edit: Nevermind, it might have just been the library I’m using. Seems like I don’t need an offset at all.
-
I might have missed some other post, but is there a bigger writeup on this project of yours? As to the specifics of digging up such information, take a look at ECFS - https://github.com/elfmaster/ecfs
-
I might have missed some other post, but is there a bigger writeup on this project of yours?
I’m afraid not, at least for the debugger subproject. This is the context. The debugger would fit in two ways:
- Since I have a GUI maker, I can try to use it to make a graphical debugger. (Ideally, allowing custom visualizations created for each new debugging task.)
- A debugger/editor would be useful for making and editing [Flpc]((github.com/asrp/flpc) or similar. I want to be able to quickly customize the debugger to also be usable as an external Flpc debugger (instead of just a C debugger). In fact, it’d be nice if I could evolve the debugger and target (=interpreter) simultaneously.
Although I’m mostly thinking of using it for the earlier stages of development. Even though I should already be past that stage, if I can (re)make that quickly, I’ll be more inclined to try out major architectural changes. And also add more functionality in C more easily.
Ideally, the debugger would also be an editor (write a few instructions, set
SIGTRAP, run, write a few more instructions, etc; write some other values to memory here and there). But maybe this is much more trouble than its worth.Your senseye program might be relevant depending on how customizable (or live customizable) the UI is. The stack on which its built is completely unknown to me. Do you have videos/posts where you use it to debug and/or find some particular piece of information?
As to the specifics of digging up such information, take a look at ECFS - https://github.com/elfmaster/ecfs
I have to say, this looks really cool. Although in my case, I’m expecting cooperation from the target being debugged.
Hopefully I will remember this link if I need something like that later on.
-
I have to say, this looks really cool. Although in my case, I’m expecting cooperation from the target being debugged.
My recommendation, coolness aside, for the ECFS part is that Ryan is pretty darn good with the ugly details of ELF and his code and texts are valuable sources of information on otherwise undocumented quirks.
Your senseye program might be relevant depending on how customizable (or live customizable) the UI is. The stack on which its built is completely unknown to me. Do you have videos/posts where you use it to debug and/or find some particular piece of information?
I think the only public trace of that is https://arcan-fe.com/2015/05/24/digging-for-pixels/ but it only uses a fraction of the features. The cases I use it for on about a weekly basis touch upon materials that are NDAd.
I have a blogpost coming up on how the full stack itself map into debugging and what the full stack is building towards, but the short short (yet long, sorry for that, the best I could do at the moment) version:
Ingredients:
Arcan is a display server - a poor word for output control, rendering and desktop IPC subsystem. The IPC subsystem is referred to as SHMIF. It also comes with a mid level client API: TUI which roughly correlates to ncurses, but with more desktop:y featureset and sidesteps terminal protocols for better window manager integration.
The SHMIF IPC part that is similar to a ‘Window’ in X is referred to as a segment. It is a typed container comprised of one big block (video frame), a number of small chunked blocks (audio frames), two ring buffers as input/output queue that carry events and file descriptors.
Durden act a window manager (Meta-UI).This mostly means input mapping, configuration tracking, interactive data routing and window layouting.
Senseye comes in three parts. The data providers, sensors, that have some means of sampling with basic statistics (memory, file, ..) which gets forwarded over SHMIF to Durden. The second part is analysis and visualization scripts built on the scripting API in Arcan. Lastly there are translators that are one-off parsers that take some incoming data from SHMIF, parses it and renders some human- useful human- level output, optionally annotated with parsing state metadata.
Recipe:
A client gets a segment on connection, and can request additional ones. But the more interesting scenario is that the WM (durden in this case) can push a segment as a means of saying ‘take this, I want you to do something with it’ and the type is a mapping to whatever UI policy that the WM cares about.
One such type is Debug. If a client maps this segment, it is expected to populate it with whatever debugging/troubleshooting information that the developer deemed relevant. This is the cooperative stage, it can be activated and deactivated at runtime without messing with STDERR and we can stop with the printf() crap.
The thing that ties it all together - if a client doesn’t map a segment that was pushed on it, because it doesn’t want to or already have one, the shmif-api library can sneakily map it and do something with it instead. Like provide a default debug interface preparing the process to attach a debugger, or activate one of those senseye sensors, or …
Hierarchical dynamic debugging, both cooperative and non-cooperative, bootstrapped by the display server connection - retaining chain of trust without a sudo ptrace side channel.
Here’s a quick PoC recording: https://youtu.be/yBWeQRMvsPc where a terminal emulator (written using TUI) exposes state machine and parsing errors when it receives a “pushed” debug window.
So what I’m looking into right now is writing the “fallback” debug interface, with some nice basics, like stderr redirect, file descriptor interception and buffer editing, and a TUI for lldb to go with it ;-)
The long term goal for all this is “every byte explained”, be able to take something large (web browser or so) and have the tools to sample, analyse, visualise and intercept everything - show that the executing runtime is much more interesting than trivial artefacts like source code.
-
Thanks! After reading this reply, I’ve skimmed your lastest post submitted here and on HN. I’ve added it to my reading list to considered more carefully later.
I don’t fully understand everything yet but get the gist of it for a number of pieces.
I think the only public trace of that is https://arcan-fe.com/2015/05/24/digging-for-pixels/ but it only uses a fraction of the features.
Thanks, this gives me a better understanding. I wouldn’t minding seeing more examples like this, even if contrived.
In my case I’m not (usually) manipulating (literal) images or video/audio streams though. Do you think your project would be very helpful for program state and execution visualization? I’m thinking of something like Online Python Tutor. (Its sources is available but unfortunately everything is mixed together and its not easy to just extract the visualization portion. Plus, I need it to be more extensible.)
For example, could you make it so that you could manually view the result for a given user-input width, then display the edges found (either overlayed or separately) and finally after playing around with it a bit (and possibly other objective functions than edges), automatically find the best width as show in the video? (And would this be something that’s easy to do?) Basically, a more interactive workflow.
The thing that ties it all together - if a client doesn’t map a segment that was pushed on it, because it doesn’t want to or already have one, the shmif-api library can sneakily map it and do something with it instead.
Maybe this is what you already meant here and by your “fallback debug interface” but how about having a separate process for “sneaky mapping”? So SHMIF remains a “purer” IPC but you can an extra process in the pipeline to do this kind of mapping. (And some separate default/automation can be toggled to have it happen automatically.)
Hierarchical dynamic debugging, both cooperative and non-cooperative, bootstrapped by the display server connection - retaining chain of trust without a sudo ptrace side channel.
Here’s a quick PoC recording: https://youtu.be/yBWeQRMvsPc where a terminal emulator (written using TUI) exposes state machine and parsing errors when it receives a “pushed” debug window.
Very nice! Assuming I understood correctly, this takes care of the extraction (or in your architecture, push) portion of the debugging
-
Just poke me if you need further clarification.
For example, could you make it so that you could manually view the result for a given user-input width, then display the edges found (either overlayed or separately) and finally after playing around with it a bit (and possibly other objective functions than edges), automatically find the best width as show in the video? (And would this be something that’s easy to do?) Basically, a more interactive workflow.
The real tool is highly interactive, it’s the basic mode of operation, it’s just the UI that sucks and that’s why it’s being replaced with Durden that’s been my desktop for a while now. This video shows a more interactive side: https://www.youtube.com/watch?v=WBsv9IJpkDw Including live sampling of memory pages (somewhere around 3 minutes in).
Maybe this is what you already meant here and by your “fallback debug interface” but how about having a separate process for “sneaky mapping”? So SHMIF remains a “purer” IPC but you can an extra process in the pipeline to do this kind of mapping. (And some separate default/automation can be toggled to have it happen automatically.)
It needs both, I have a big bag of tricks for the ‘in process’ part, and with YAMA and other restrictions on ptrace these days the process needs some massage to be ‘external debugger’ ready. Though some default of “immediately do this” will likely be possible.
I’ve so far just thought about it interactively with the sortof goal that it should be, at most, 2-3 keypresses from having a window selected to be digging around inside it’s related process no matter what you want to measure or observe. https://github.com/letoram/arcan/blob/master/src/shmif/arcan_shmif_debugif.c ) not finished by any stretch binds the debug window to the TUI API and will present a menu.
Assuming I understood correctly, this takes care of the extraction (or in your architecture, push) portion of the debugging
Exactly.
-
Thanks. So I looked a bit more into this.
I think the most interesting part for me at the moment is the disassembly.
I tried to build it just to see. I eventually followed these instructions but can’t find any Senseye related commands in any menu in Durden (global or target).
I think I managed to build senseye/senses correctly.
Nothing obvious stands out in tools. I tried both symlinks
/path/to/durden/durden/tools/senseye/senseye /path/to/durden/durden/tools/senseye/senseye.luaand
/path/to/durden/durden/tools/senseye /path/to/durden/durden/tools/senseye.luaHere are some other notes on the build process
LibdrmOn my system, the include -I/usr/include/libdrm and linker flag -ldrm are needed. I don’t know cmake so don’t know where to add them. (I manually edited and ran the commands
make VERBOSE=1was running to get around this.)I had to replace some
DurdenCODEC_*withAV_CODEC_*Initially Durden without -p /path/to/resources would not start saying some things are
Senseyebroken. I can’t reproduce it anymore.cmake -DARCAN_SOURCE_DIR=/path/to/src ../sensescomplains about
ARCAN_TUI_INCLUDE_DIRandARCAN_TUI_LIBRARYbeing not found:
CapstoneMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: ARCAN_TUI_INCLUDE_DIRI eventually installed Arcan instead of just having it built and reached this error
No rule to make target 'capstone/lib/libcapstone.a', needed by 'xlt_capstone'.I symlinked
Odd crashescapstone/lib64tocapstone/libto get around this.Sometimes, Durden crashed (or at least exited without notice) like when I tried changing resolution from inside.
Here’s an example:
Debug windowImproper API use from Lua script: target_disphint(798, -2147483648), display dimensions must be >= 0 stack traceback: [C]: in function 'target_displayhint' /path/to/durden/durden/menus/global/open.lua:80: in function </path/to/durden/durden/menus/global/open.lua:65> Handing over to recovery script (or shutdown if none present). Lua VM failed with no fallback defined, (see -b arg).I did get target->video->advanced->debug window to run though.
-
I’d give it about two weeks before running senseye as a Durden extension is in a usable shape (with most, but not all features from the original demos).
A CMake FYI - normally you can patch the CMakeCache.txt and just make. Weird that it doesn’t find the header though, src/platform/cmake/FindGBMKMS.cmake quite explicitly looks there, hmm…
The old videos represent the state where senseye could run standalone and did its own window management. For running senseye in the state it was before I started breaking/refactoring things the setup is a bit different and you won’t need durden at all. Just tested this on OSX:
- Revert to an old arcan build ( 0.5.2 tag) and senseye to the tag in the readme.
- Build arcan with -DVIDEO_PLATFORM=sdl (so you can run inside your normal desktop) and -DNO_FSRV=On so the recent ffmpeg breakage doesn’t hit (the AV_CODEC stuff).
- Build the senseye senses like normal, then arcan /path/to/senseye/senseye
Think I’ve found the scripting error, testing when I’m back home - thanks.
The default behavior on scripting error is to shutdown forcibly even if it could recover - in order to preserve state in the log output, the -b arguments lets you set a new app (or the same one) to switch and migrate any living clients to, arcan -b /path/to/durden /path/to/durden would recover “to itself”, surprisingly enough, this can be so fast that you don’t notice it has happened :-)
-
Thanks, with these instructions I got it compiled and running. I had read the warning in senseye’s readme but forgot about it after compiling the other parts.
I’m still stumbling around a bit, though that’s what I intended to do.
So it looks like the default for
sense_memis to not interrupt the process. I’m guessing the intended method is to use ECFS to snapshot the process and view later. But I’m actually trying to live view and edit a process.Is there a way to view/send things through the IPC?
From the wiki:
The delta distance feature is primarily useful for polling sources, like the mem-sense with a refresh clock. The screenshot below shows the alpha window picking up on a changing byte sequence that would be hard to spot with other settings.
Didn’t quite understand this example. Mem diff seems interesting in general.
For example, I have a program that changes a C variable’s value every second. Assuming we don’t go read the ELF header, how can senseye be used to find where that’s happening?
From another part of the wiki
and the distinct pattern in the point cloud hints that we are dealing with some ASCII text.
This could use some more explanation. How can you tell its ASCII from just a point cloud??
Minor questions/remarkNot urgent in any way
- Is there a way to start the process as a child so ./sense_mem needs less permissions?
- Is there a way to view registers?
Compiling senseye without installing Arcan with
cmake -DARCAN_SOURCE_DIR=still gives errors.I think the first error was about undefined symbols that were in platform/platform.h (
arcan_aobj_idandarcan_vobj_id).I can try to get the actual error message again if that’s useful.
-
Thanks, with these instructions I got it compiled and running. I had read the warning in senseye’s readme but forgot about it after compiling the other parts. I’m still stumbling around a bit, though that’s what I intended to do.
From the state you’re seeing it, it is very much a research project hacked together while waiting at airports :-) I’ve accumulated enough of a idea to distill it into something more practically thought together - but not there quite yet.
Is there a way to view/send things through the IPC?
At the time it was written, I had just started to play with that (if you see the presentation slides, that’s the fuzzing bit, the actual sending works very much like a clipboard paste operation), the features are in the IPC system now, not mapped into the sensors though.
So it looks like the default for sense_mem is to not interrupt the process. I’m guessing the intended method is to use ECFS to snapshot the process and view later. But I’m actually trying to live view and edit a process.
yeah, sense_mem was just getting the whole “what does it take to sample / observe process memory without poking it with ptrace etc. Those controls and some other techniques are intended to be bootstrapped via the whole ipc-system in the way I talked about earlier. That should kill the privilege problem as well.
Didn’t quite understand this example. Mem diff seems interesting in general.
The context menu for a data window should have a refresh clock option. If that’s activated, it will re-sample the current page and mark which bytes changed. Then the UI/shader for alpha window should show which bytes those are.
For example, I have a program that changes a C variable’s value every second. Assuming we don’t go read the ELF header, how can senseye be used to find where that’s happening?
The intended workflow was something like “dig around in memory, look at projections or use the other searching tools to find data of interest” -> attach translators -> get symbolic /metadata overview.
and the distinct pattern in the point cloud hints that we are dealing with some ASCII text. This could use some more explanation. How can you tell its ASCII from just a point cloud??
See the linked videos on “voyage of the reverse” and the recon 2014 video of “cantor dust”, i.e. a feedback loop of projections + training + experimentation. The translators was the tool intended to make the latter stage easier.
-
I’d give it about two weeks before running senseye as a Durden extension is in a usable shape (with most, but not all features from the original demos).
A CMake FYI - normally you can patch the CMakeCache.txt and just make. Weird that it doesn’t find the header though, src/platform/cmake/FindGBMKMS.cmake quite explicitly looks there, hmm…
The old videos represent the state where senseye could run standalone and did its own window management. For running senseye in the state it was before I started breaking/refactoring things the setup is a bit different and you won’t need durden at all. Just tested this on OSX:
- Revert to an old arcan build ( 0.5.2 tag) and senseye to the tag in the readme.
- Build arcan with -DVIDEO_PLATFORM=sdl (so you can run inside your normal desktop) and -DNO_FSRV=On so the recent ffmpeg breakage doesn’t hit (the AV_CODEC stuff).
- Build the senseye senses like normal, then arcan /path/to/senseye/senseye
Think I’ve found the scripting error, testing when I’m back home - thanks.
The default behavior on scripting error is to shutdown forcibly even if it could recover - in order to preserve state in the log output, the -b arguments lets you set a new app (or the same one) to switch and migrate any living clients to, arcan -b /path/to/durden /path/to/durden would recover “to itself”, surprisingly enough, this can be so fast that you don’t notice it has happened :-)
-
-
-
-
-
-
-
-
-
-
If you are looking for references on debuggers then the book How Debuggers Work may be helpful.
-
Your response is much better than the original article (which has lowered my opinion of the ACM) but I do have one correction.
Tomasulo’s algorithm, the first implementation of out-of-order execution, predates the C language by a few years.-
Your response…
Quick note: I did not write this, if that’s who you’re addressing. The author is Robert Graham.
-
-
-
Since I’m graduated, I have loads of free time now before starting my new job in a few weeks :)
As an exercise to learn D, I’ve been working on reimplementing MPD in D (https://github.com/charles-l/mop). I am really liking it so far - I feel far more secure in the correctness of my code than with C or C++, and the compile-time code generation is the most flexible I’ve seen for a systems-level language.
I’m writing some bindings for the Zumo 32u4 robot for microscheme (https://github.com/charles-l/zumo-32u4-ms), and hoping to get most of the sensors working this week.
Also, I’ve been messing around with J, hoping to learn it well enough to use it sort of as a desktop calculator on steriods for writing throwaway scripts. I wrote a Kalman filter with J last week, and will probably write an Extended Kalman filter or something similar this week.
I’ll probably also work a bit more on my graphics engine (https://github.com/charles-l/ugg) and possibly a new language project.
-
All of that sounds really cool. :) On desktop calculator, anyone interested in calculators mixed with programming should check out Frink. It’s a neat, little project.
-
-
That’s the main reason I kept it. I was thinking having common units in mainstream languages would be a good idea. Frink would be a reference if I ever tried to add it to my own.
-
-
-
-
-
Personal - I spent most of this weekend working on a fork of the ODMA reference implementation source. All of the code compiles but the test application throws an MFC assertion failure when run. I have decided this is nature’s way of encouraging me to learn how to use a native code debugger.
Work - I have the usual mix of user support issues and DMS administrative tasks. I will be spending a lot of this week covering for other people who are on vacation.
-
I have started hacking on the ODMA sample code. My eventual goal is a Notepad++ plugin for the document management system that I use at work.
Work - Reporting bugs to our DMS (Document Management System) vendor (2 so far this month), the usual support tickets, and maintenance for my employer’s node on a blockchain network.