The previous concept reapplied from software was the black box analysis technique. The technique metaphorically places something inside a black box, which signifies avoidance of direct scrutiny or even identification. The something's effects are examined instead, thereby circumventing the interference or the labor of knowing the something and its inner workings. The analysis proceeds through the factual details of various interactions between the something and its environment.
It's highly relevant to the goal of objective testing, because it avoids prejudices. The act of inspection is entangled in the inspector's slanted perspective, while black box tests compare clear-cut outcomes to uninfluenced expectations. If external outcomes don't satisfy neutral and sensible criteria then the something should be reevaluated, regardless of who/what it is and the characteristics it supposedly has within.
Beyond black boxes, the topic of testing software includes another broadly useful concept: smoke tests. These are rapid, shallow, preliminary, unmistakable checks that the software is minimally serviceable. The name comes from the analogy of activating electronic equipment and just seeing if it smokes. A smoke test of software runs the smallest tasks. Can it start? Can it locate the software that it teams up with? Can it load its configuration? Can it produce meaningful output at all?
No specialized expertise is necessary to notice that smoke tests are vital but also laughably inadequate. Since the software must pass much more rigorous tests, it's logical to question why smoke tests are worthwhile to perform more than once on the same software. However, the bare fact is that software seldom stays the same, especially in the middle of furious development. Thus the worth of smoke tests is more for quickly determining when a recent modification is gravely problematic. A malfunctioning smoke test implies the need to reconsider the recent modification and rectify it—probably very soon in order to prevent delays in the overall schedule.
The surprise is that smoke tests resemble a mental tactic that shows up in various informal philosophizing. Like software developers who screen their attempts with smoke tests and then promptly fix and retry in the event of failure, a follower of a belief may repeatedly rethink its specifics until it's acceptable according to the equivalent of a smoke test. In essence the follower has a prior commitment to a conclusion, which they purposely reshape so that it at least doesn't "smoke". This tactic greatly differs from carefully proposing a tentative claim after collecting well-founded corroboration. And it differs from the foundation of productive debate: the precondition that the debaters' arguments are like orderly chains from one step to the next, not like lumps of clay that continually transform to evade objections.
As might be expected, the smoke test tactic easily leads to persistent misunderstandings about aims. The unambitious aim of the tactic is a pruned belief that isn't flagrantly off-base, not a pristine belief that's most likely to be accurate. A few belief smoke tests are absurdity, contradiction with solidly established information, violation of common contemporary ethics, and so forth. (The changes might qualify as retcons.) Before they show the candor to concede that their aim is a treasured belief that isn't transparently wrong, rather than the novel belief that's plausibly right, they're mired in a loop of mending belief by trial and error.
They may justify the tactic by saying, "Of course I can't profess the most uncomplicated, unswerving variant of my belief. I know that variant can't be correct. It would be too [absurd, barbaric, intolerant, naive, infeasible, bizarre, self-contradictory]. I use my best understanding to strengthen the weak points that ring false. Doesn't everyone? Why's that a reason for criticism?"
This rationale is persuasive; to revise beliefs over time is no shortcoming. The telling difference is that everyone else isn't using the tactic on beliefs portrayed as complete, authoritative, correct, and self-supporting. It presents two issues in that case. First, why would the belief have been communicated in such a way that the recipients need to make fine-grained clarifications for the sake of succeeding at smoke tests—which are exceedingly basic, after all? Second, once someone has begun increasingly reworking the original belief to comply with their sense of reasonableness, when does the belief itself stop being a recognizable, beneficial contributor to the result? Is it not a bad sign when something requires numerous manual interventions, replacement of parts, and gentle handling, or else it swiftly proceeds to belch embarrassing smoke?
Showing posts with label Software development Observations. Show all posts
Showing posts with label Software development Observations. Show all posts
Monday, September 07, 2015
Sunday, August 02, 2015
black boxes blocking baseless bias
Considering the proportion of time filled by a full-time career, its thought patterns carve deep grooves. Hence the blog winds up with entries musing on the wider application of software patterns like, say, competing structures. In a software project, diverse structures of data and code could all be part of doable solutions. But the project allows only one solution. Not all of these structures have equal quality, so a competition is appropriate. Meanwhile in the philosophical domain and elsewhere, humans contrive diverse mental structures for the "project" of thinking and acting within their puzzling realities. And it shouldn't be verboten for these structures of uneven quality to fairly compete.
That's the toughest obstacle in practice: defining and applying legitimately equitable standards of comparison. Whenever evaluators have decided beforehand that the structures they endorse will be superior, then their tendency is to choose and distort the standards to assure it. The ones committed to candor readily admit this; even better, if they're confident then they welcome offers of separate reviews that will validate the credibility of their own.
Luckily, the ceaseless struggle to approach ideas with less partisanship has another pattern back in the technological domain. The common black box technique refers to analyzing something purely via the stuff entering and exiting it. Knowledge about the thing, and its contents, is excluded for whatever reason. Conceptually, the thing is hidden inside a black box with little holes for stuff to pass through. On a diagram, multiple arrows go to and from the box, but nothing is written in the box except its name. As a side note, a representation of a single, huge, unexamined thing containing miscellaneous parts, such as an external computer network, might have been drawn as a bumpy cloud to emphasize its vague "shape" and "size".
The black box analysis is simplified and undeniably easier to manage as a consequence. Sometimes, depending on the task, the thing's innards are mostly off-topic. To smoothly interact with the thing, the more crucial details are what agreed-upon stuff will come out (or occur) after agreed-upon stuff goes in. Without condensed black box abstractions, the modern industrial age of specialized, interchangeable technology would be infeasible. Everyone would need to know an excessive amount about the individually complex pieces merely to construct a functioning whole. This is an equally essential ingredient of software. With published protocols and data formats, software can handle other software as black box peers which accept and emit lucid messages. Broad classes of compliant software can profitably cooperate.
Overall, an extensive black box description is invaluable for something that's largely unknown—by design or by circumstance. In contrast, the value of a black box description for something that's largely known is less intuitive. It hinges on the recognition that too-close familiarity with something might build a deceptive or incomplete impression of its satisfactoriness. When the something is software, it's only logical that its industrious writer is unaware of its oversights, else they wouldn't have written the oversights. At writing time, they may have framed their solution too narrowly to enclose the project's range of subtleties. Later, their ongoing frame goes on to prevent them from imagining tests capable of exposing the cramped, inadequate boundaries.
Mistakes of oversight are rivaled by occasionally embarrassing mistakes of "transcription": the writer failed to faithfully encode their original intent. They wanted to read memory location Q but they wrote code that reads J. Once again, it's only logical that such mistakes wouldn't survive if the writer's own firsthand experience caught every gaffe they introduced. They may have been distracted. Depressingly often, disorganization gradually accumulates in the code segment. Or, in a less forgivable offense, it's confusingly expressed from the outset. As a result, although they're staring directly at a mistake, they're distracted by the onerous strain of deciphering and tracking the bigger picture.
Less specifically, the sizable value of black box analysis for a largely known something lies in cross-checking the fallible judgments of "insiders" about that something. Placing it in a black box counteracts the hypothetical shortcomings of the insiders' entanglement. It includes putting aside comprehensive information about something's unique identity and full set of characteristics, and putting aside other connections/relationships, and putting aside appeal/repulsiveness. It's the candid, untainted estimation of whether the something's observed "footprints" match levelheaded expectations in pertinent contexts. The writer's admirable pride of craftsmanship doesn't attest that the supposedly finished software unit operates acceptably in all probable cases.
This practice's basic features are visible throughout innumerable domains, though with varied titles. It chimes with the "blinding" of subjects in experiments and surveys of customer tastes. (In an unusually palpable manifestation of the metaphor, part of the blinding procedure might employ nondescript, opaque boxes.) Blinding forces them to assess the sample with the sole attribute they can sense. From their view, the sample's source is in a black box. A second example is the services of an editor. They can approve and/or modify sections of a draft document according to the unprepared reactions it elicits in them. Unlike the submitter, they aren't an "expert" at knowing what it's meant to convey. They don't feel the submitter's strong sentimental attachments. They have a greater chance of encountering the draft itself. Where the editor is concerned, the labor behind it doesn't affect their revisions. The draft came out of a black box.
A third example is in the same theme, albeit more cerebral. It's the strategy of, after a long session of work on a preliminary creation, reserving time away before revisiting it. In the heart of the session, the creation is summarizing a portion of the creator's stream of consciousness. Therefore the contemporaneous brain activity grants them the perfect ability to effortlessly compensate for the creation's ambiguities and awkward aspects. To them in that instant, the creation's "seamless" substance and beauty are impossible to miss. When they return, their brain's state isn't enmeshed with the creation. They take a fresh look at its pluses and minuses in isolation. This is akin to the advice of not transforming a late-night brainstorm into irreversible actions until pondering it next morning. Interestingly, the something in the black box is the past configuration of the brain currently reexperiencing that past configuration's product, i.e. the creation/brainstorm. The critical difference is that the product isn't rubber-stamped due to where it came from—whose brain it rippled out of. No, the caliber of the product discloses the worthiness of whatever produced it, in this instance a past brain configuration. (It might be uncomplimentary. "My brain was really mesmerized by that tangent, but this is unusable nonsense.")
Despite its encouragingly widespread and timeless scope, black box-style thinking is a supplemental tool with inherent limits. It's for temporarily redirecting attention to the external symptoms of something's presence. Its visual counterpart is a sketch of a silhouette. It doesn't capture something's essence. It's not an explanation; on its own, a lengthy historical listing doesn't reliably predict responses to novel situations.
The epitome of an area dominated by these caveats is human conduct. Without question, the brain's convoluted character precludes painless black box analysis for rigorously unraveling how it runs. It exhibits context-dependent overrides of overrides of overrides...or it might not. Trends discovered during good tempers may have little relation to bad tempers. Or mannerisms connected to one social group may have little relation to mannerisms connected to contrasting social groups. Or a stranger switches among several conscious (or unconscious) guises, aimed at selectively steering the verdicts of unacquainted onlookers. The stranger is in a black box to the onlookers. The guises are collections of faked signals chosen to misinform the onlookers' analysis of that black box.
Caveats notwithstanding, entire societies heavily regulate members through black boxes of human conduct. (As a popular song from the early nineties famously didn't proclaim, "Here we are now: in containers.") Members are efficiently pigeonholed by unsophisticated facts about their deeds. In the society, facts of that type serve as decisive announcements of the member's inner nature. So, members who wish to be seen a certain way are obliged to adhere to the linked mandates. No extra particulars about them are accounted for. For this purpose, they're in a black box. It appears callous at first glance, but it exemplifies the earlier statements about the value of shortcuts for working with something that's largely unknown. When societies reach massive scales, it's impossible for members to obtain penetrating awareness of every other member. Like before, black box understandings ease interactions with scarce information about either party, becauses the pair can foresee what will transpire between them.
Furthermore, black box analysis of human conduct shares the advantages stated earlier for inspecting something that's largely known. The effectiveness is lowered by the caveats of this area but not eliminated altogether. It's more than adequate for imposing sharp, sensible thresholds on other findings. "If I didn't know them as well as I do, and they acted the same as they have in the situations I know of, would there be a disparity in how I esteem them? If there is, do I have a well-founded excuse for it? At some point, my firmest convictions about who they are should be aligning to some degree with their acts...."
That's the toughest obstacle in practice: defining and applying legitimately equitable standards of comparison. Whenever evaluators have decided beforehand that the structures they endorse will be superior, then their tendency is to choose and distort the standards to assure it. The ones committed to candor readily admit this; even better, if they're confident then they welcome offers of separate reviews that will validate the credibility of their own.
Luckily, the ceaseless struggle to approach ideas with less partisanship has another pattern back in the technological domain. The common black box technique refers to analyzing something purely via the stuff entering and exiting it. Knowledge about the thing, and its contents, is excluded for whatever reason. Conceptually, the thing is hidden inside a black box with little holes for stuff to pass through. On a diagram, multiple arrows go to and from the box, but nothing is written in the box except its name. As a side note, a representation of a single, huge, unexamined thing containing miscellaneous parts, such as an external computer network, might have been drawn as a bumpy cloud to emphasize its vague "shape" and "size".
The black box analysis is simplified and undeniably easier to manage as a consequence. Sometimes, depending on the task, the thing's innards are mostly off-topic. To smoothly interact with the thing, the more crucial details are what agreed-upon stuff will come out (or occur) after agreed-upon stuff goes in. Without condensed black box abstractions, the modern industrial age of specialized, interchangeable technology would be infeasible. Everyone would need to know an excessive amount about the individually complex pieces merely to construct a functioning whole. This is an equally essential ingredient of software. With published protocols and data formats, software can handle other software as black box peers which accept and emit lucid messages. Broad classes of compliant software can profitably cooperate.
Overall, an extensive black box description is invaluable for something that's largely unknown—by design or by circumstance. In contrast, the value of a black box description for something that's largely known is less intuitive. It hinges on the recognition that too-close familiarity with something might build a deceptive or incomplete impression of its satisfactoriness. When the something is software, it's only logical that its industrious writer is unaware of its oversights, else they wouldn't have written the oversights. At writing time, they may have framed their solution too narrowly to enclose the project's range of subtleties. Later, their ongoing frame goes on to prevent them from imagining tests capable of exposing the cramped, inadequate boundaries.
Mistakes of oversight are rivaled by occasionally embarrassing mistakes of "transcription": the writer failed to faithfully encode their original intent. They wanted to read memory location Q but they wrote code that reads J. Once again, it's only logical that such mistakes wouldn't survive if the writer's own firsthand experience caught every gaffe they introduced. They may have been distracted. Depressingly often, disorganization gradually accumulates in the code segment. Or, in a less forgivable offense, it's confusingly expressed from the outset. As a result, although they're staring directly at a mistake, they're distracted by the onerous strain of deciphering and tracking the bigger picture.
Less specifically, the sizable value of black box analysis for a largely known something lies in cross-checking the fallible judgments of "insiders" about that something. Placing it in a black box counteracts the hypothetical shortcomings of the insiders' entanglement. It includes putting aside comprehensive information about something's unique identity and full set of characteristics, and putting aside other connections/relationships, and putting aside appeal/repulsiveness. It's the candid, untainted estimation of whether the something's observed "footprints" match levelheaded expectations in pertinent contexts. The writer's admirable pride of craftsmanship doesn't attest that the supposedly finished software unit operates acceptably in all probable cases.
This practice's basic features are visible throughout innumerable domains, though with varied titles. It chimes with the "blinding" of subjects in experiments and surveys of customer tastes. (In an unusually palpable manifestation of the metaphor, part of the blinding procedure might employ nondescript, opaque boxes.) Blinding forces them to assess the sample with the sole attribute they can sense. From their view, the sample's source is in a black box. A second example is the services of an editor. They can approve and/or modify sections of a draft document according to the unprepared reactions it elicits in them. Unlike the submitter, they aren't an "expert" at knowing what it's meant to convey. They don't feel the submitter's strong sentimental attachments. They have a greater chance of encountering the draft itself. Where the editor is concerned, the labor behind it doesn't affect their revisions. The draft came out of a black box.
A third example is in the same theme, albeit more cerebral. It's the strategy of, after a long session of work on a preliminary creation, reserving time away before revisiting it. In the heart of the session, the creation is summarizing a portion of the creator's stream of consciousness. Therefore the contemporaneous brain activity grants them the perfect ability to effortlessly compensate for the creation's ambiguities and awkward aspects. To them in that instant, the creation's "seamless" substance and beauty are impossible to miss. When they return, their brain's state isn't enmeshed with the creation. They take a fresh look at its pluses and minuses in isolation. This is akin to the advice of not transforming a late-night brainstorm into irreversible actions until pondering it next morning. Interestingly, the something in the black box is the past configuration of the brain currently reexperiencing that past configuration's product, i.e. the creation/brainstorm. The critical difference is that the product isn't rubber-stamped due to where it came from—whose brain it rippled out of. No, the caliber of the product discloses the worthiness of whatever produced it, in this instance a past brain configuration. (It might be uncomplimentary. "My brain was really mesmerized by that tangent, but this is unusable nonsense.")
Despite its encouragingly widespread and timeless scope, black box-style thinking is a supplemental tool with inherent limits. It's for temporarily redirecting attention to the external symptoms of something's presence. Its visual counterpart is a sketch of a silhouette. It doesn't capture something's essence. It's not an explanation; on its own, a lengthy historical listing doesn't reliably predict responses to novel situations.
The epitome of an area dominated by these caveats is human conduct. Without question, the brain's convoluted character precludes painless black box analysis for rigorously unraveling how it runs. It exhibits context-dependent overrides of overrides of overrides...or it might not. Trends discovered during good tempers may have little relation to bad tempers. Or mannerisms connected to one social group may have little relation to mannerisms connected to contrasting social groups. Or a stranger switches among several conscious (or unconscious) guises, aimed at selectively steering the verdicts of unacquainted onlookers. The stranger is in a black box to the onlookers. The guises are collections of faked signals chosen to misinform the onlookers' analysis of that black box.
Caveats notwithstanding, entire societies heavily regulate members through black boxes of human conduct. (As a popular song from the early nineties famously didn't proclaim, "Here we are now: in containers.") Members are efficiently pigeonholed by unsophisticated facts about their deeds. In the society, facts of that type serve as decisive announcements of the member's inner nature. So, members who wish to be seen a certain way are obliged to adhere to the linked mandates. No extra particulars about them are accounted for. For this purpose, they're in a black box. It appears callous at first glance, but it exemplifies the earlier statements about the value of shortcuts for working with something that's largely unknown. When societies reach massive scales, it's impossible for members to obtain penetrating awareness of every other member. Like before, black box understandings ease interactions with scarce information about either party, becauses the pair can foresee what will transpire between them.
Furthermore, black box analysis of human conduct shares the advantages stated earlier for inspecting something that's largely known. The effectiveness is lowered by the caveats of this area but not eliminated altogether. It's more than adequate for imposing sharp, sensible thresholds on other findings. "If I didn't know them as well as I do, and they acted the same as they have in the situations I know of, would there be a disparity in how I esteem them? If there is, do I have a well-founded excuse for it? At some point, my firmest convictions about who they are should be aligning to some degree with their acts...."
Tuesday, July 07, 2015
competing structures
Lately I've been describing examples of ideas that overlap between my software career and my philosophical positions. The foremost consequence is the thorough puncturing of information's abstract mystique. First, the traceable meaningfulness of information is rooted in the corresponding work performed by teams of computers and humans; conversely, the traceable meaning of the work is shown in the corresponding transformations of information that the work achieved. This principle underscores that information is tied to concrete efforts, and it doesn't arise out of nothing or exist independently. Second, when a computer performs information work, the humdrum process backing it is less like mystical transfiguration than like sending water through a dauntingly intricate maze of pipes, as countless synchronized valves rapidly toggle. This principle underscores that neither information nor its changes have nonphysical foundations.
The third example of overlap is the competition among structures to be used in software projects. Projects have more than one hypothetical solution. A solution contains particular structures to represent and store the targeted information, e.g. a short alphanumeric sequence for a license plate. Additionally, the solution has a structure for the code to manipulate the information, a structure which joins together separate actions for differing circumstances (an algorithm governing algorithms). Thus, depending on the analyst's choices, the total solution houses information in varying discrete sets of structures. Each set might be functional and intelligible. Nevertheless, the structures could have serious faults relative to one another: redundant, complicated, circuitous, simplistic, disorganized, bewildering, fragile. What's worse, frequently the problems aren't apparent until more time passes, at which point the structures need to be delicately replaced or reshaped. Not all of the prospective structures that are doable for the project are equally faultless and prudent. And this is reconfirmed once shortsighted structures proceed to collide with challenging realities. ("I wish these modifications had been anticipated before the structure of this code was chosen.")
Instructive parallels to the principle of competing structures aren't hard to find outside of software. In so many subtle, open-ended contexts, there isn't a uniquely correct conclusion strictly reachable through systematic steps of deductive logic. As a result, humans end up with widely divergent "mental structures" as they attempt to grasp their confusing experiences. While they don't need to turn those structures into effective software, they do need to apply these structures of interpretation to bring order to their thoughts and acts. If they're considered sane, a lot of their adopted structures probably have at least a little coherency and accuracy. As disparate as the structures might be, obviously each is good enough in the adopter's estimation. The differences might even be superficial on closer examination. After all, as much as possible the entire group of structures should be accommodating constraints that are universal: the crucial details asserted by reliable evidence and/or by other, prior. well-established structures.
Regardless, again like the technological structures in software projects, the potential for numerous candidates does not imply that all have identical quality as judged by every standard. An organizing structure can be possible without attaining a competitive level of plausibility. Although the normal complexities of existence might not dictate an obvious and definitive singular structure, intense critique casts doubt on some candidate structures more than others. For instance, belief structures seem more dubious after calling for repeated drastic revisions, i.e. retcons. So are structures that propose "backstage" causes which happen to be almost completely undetectable by impartial investigators. Structures that avoid claiming unbounded certainty merely earn a ribbon for sincere participation in the competition of realistic ideas, not instantly gain as much credibility as the leading structures that also avoid this glaring flaw.
The criteria for ranking require great care as well. Explanatory structures should compete based on thoughtful neutral guidelines, not on indulgence of the favoritism embedded in preconceptions and preferences. Brisk disregard of a structure's failure to withstand unbiased evaluation is an error-prone strategy. Note that like items placed indifferently on the pans of a balance scale, directly measuring one structure alongside a second shouldn't be construed as close-minded disrespect toward either—provided the method of comparison in fact fair and not like a tilted scale.
Generally speaking, the principle of competing structures thrives in the commonplace domains that are unsuitable for the two extreme alternatives. These are domains where there isn't one indisputable answer, but at the same time the multitude of answers aren't of uniform worth by any means. Of course, software projects are far from the only case. For an art commission, a dazzling breadth of works would meet the bare specifications...though some might consistently evoke uncomplimentary descriptions such as insipid, garish, disjointed, derivative, slapdash, repellent, etc. Out of all the works that qualified for the commission, who would then foolishly suggest that some couldn't be shoddier than the rest, or that comparative shoddiness doesn't matter?
The third example of overlap is the competition among structures to be used in software projects. Projects have more than one hypothetical solution. A solution contains particular structures to represent and store the targeted information, e.g. a short alphanumeric sequence for a license plate. Additionally, the solution has a structure for the code to manipulate the information, a structure which joins together separate actions for differing circumstances (an algorithm governing algorithms). Thus, depending on the analyst's choices, the total solution houses information in varying discrete sets of structures. Each set might be functional and intelligible. Nevertheless, the structures could have serious faults relative to one another: redundant, complicated, circuitous, simplistic, disorganized, bewildering, fragile. What's worse, frequently the problems aren't apparent until more time passes, at which point the structures need to be delicately replaced or reshaped. Not all of the prospective structures that are doable for the project are equally faultless and prudent. And this is reconfirmed once shortsighted structures proceed to collide with challenging realities. ("I wish these modifications had been anticipated before the structure of this code was chosen.")
Instructive parallels to the principle of competing structures aren't hard to find outside of software. In so many subtle, open-ended contexts, there isn't a uniquely correct conclusion strictly reachable through systematic steps of deductive logic. As a result, humans end up with widely divergent "mental structures" as they attempt to grasp their confusing experiences. While they don't need to turn those structures into effective software, they do need to apply these structures of interpretation to bring order to their thoughts and acts. If they're considered sane, a lot of their adopted structures probably have at least a little coherency and accuracy. As disparate as the structures might be, obviously each is good enough in the adopter's estimation. The differences might even be superficial on closer examination. After all, as much as possible the entire group of structures should be accommodating constraints that are universal: the crucial details asserted by reliable evidence and/or by other, prior. well-established structures.
Regardless, again like the technological structures in software projects, the potential for numerous candidates does not imply that all have identical quality as judged by every standard. An organizing structure can be possible without attaining a competitive level of plausibility. Although the normal complexities of existence might not dictate an obvious and definitive singular structure, intense critique casts doubt on some candidate structures more than others. For instance, belief structures seem more dubious after calling for repeated drastic revisions, i.e. retcons. So are structures that propose "backstage" causes which happen to be almost completely undetectable by impartial investigators. Structures that avoid claiming unbounded certainty merely earn a ribbon for sincere participation in the competition of realistic ideas, not instantly gain as much credibility as the leading structures that also avoid this glaring flaw.
The criteria for ranking require great care as well. Explanatory structures should compete based on thoughtful neutral guidelines, not on indulgence of the favoritism embedded in preconceptions and preferences. Brisk disregard of a structure's failure to withstand unbiased evaluation is an error-prone strategy. Note that like items placed indifferently on the pans of a balance scale, directly measuring one structure alongside a second shouldn't be construed as close-minded disrespect toward either—provided the method of comparison in fact fair and not like a tilted scale.
Generally speaking, the principle of competing structures thrives in the commonplace domains that are unsuitable for the two extreme alternatives. These are domains where there isn't one indisputable answer, but at the same time the multitude of answers aren't of uniform worth by any means. Of course, software projects are far from the only case. For an art commission, a dazzling breadth of works would meet the bare specifications...though some might consistently evoke uncomplimentary descriptions such as insipid, garish, disjointed, derivative, slapdash, repellent, etc. Out of all the works that qualified for the commission, who would then foolishly suggest that some couldn't be shoddier than the rest, or that comparative shoddiness doesn't matter?
Saturday, May 30, 2015
journey to the center of the laptop
The last time I described how ideas from my software career shaped my present thinking, the topic was the interdependency between the meanings of data and code. The effective meaning of data was rooted in the details of "information systems" behind it: purposeful sequences of computer code and human labor to methodically record it, construct it, augment it, alter it, mix it with more data, etc. But the same observation could be reversed: the effective meaning (and correctness) of the information system was no more than its demonstrated transformations of data.
This viewpoint appeared to apply in other domains as well. For a wide range of candidate concepts, probing the equivalent of the concept's supporting "information system" usefully sifted its detectable meaning. How did the concept originally arise? How could the concept's definitions, verifications, and interpretations be (or fail to be) repeated and rechecked? Prospective data was discarded if it didn't have satisfactory answers to these questions; should pompous concepts face lower standards?
However, not all the software ideas were at the scale of information systems. Some knowledge illuminated the running of a single laptop. For instance, where does the laptop's computation happen? Where's the site of its mysterious data alchemy? What's the core of its "thinking"—with the precondition that this loaded term is applied purely in the loose, informal, metaphorical sense? (Note that the following will rely on simplified technological generalizations too...) The natural course of investigation is from the outside in.
This viewpoint appeared to apply in other domains as well. For a wide range of candidate concepts, probing the equivalent of the concept's supporting "information system" usefully sifted its detectable meaning. How did the concept originally arise? How could the concept's definitions, verifications, and interpretations be (or fail to be) repeated and rechecked? Prospective data was discarded if it didn't have satisfactory answers to these questions; should pompous concepts face lower standards?
However, not all the software ideas were at the scale of information systems. Some knowledge illuminated the running of a single laptop. For instance, where does the laptop's computation happen? Where's the site of its mysterious data alchemy? What's the core of its "thinking"—with the precondition that this loaded term is applied purely in the loose, informal, metaphorical sense? (Note that the following will rely on simplified technological generalizations too...) The natural course of investigation is from the outside in.
- To start with, probably everyone who regularly uses a laptop would say that the thinking takes place inside the unit. The ports around its edges for connecting audio, video, networking, generic devices, etc. are optional. These connections are great for enabling additional options to transport information to and from the laptop, but they don't enable the laptop to think. The exceptions are the battery slot and/or the power jack, which are nonetheless only providers of the raw energy consumed by the laptop's thinking.
- Similarly, it doesn't require technical training to presume that the laptop's screen, speakers, keyboard, touchpad, camera, etc. aren't where the laptop thinks. The screen may shut off to save power. The speakers may be muted. The keyboard and touchpad are replaceable methods to detect and report the user's motions. Although these accessible inputs and outputs are vital to the user's experience of the laptop, their functions are like translation rather than thinking. Either the user's actions are transported to the laptop's innards as streams of impulses, or the final outcomes of the laptop's thinking are transported back out to the user's senses.
- Consequently, the interior is a more promising space to look. Encased in the walls of the laptop, under the keyboard, behind the speakers, is a meticulously assembled collection of incredibly flat and thin parts. Some common kinds of parts are temporary memory (RAM), permanent storage (internal drives), disc drives (CD,DVD,Blu-Ray), wireless networking (WiFi). By design this group receives, holds, and sends information. Information is transported but not thought about. So the thinking must occur in the component that's on the opposite side of this group's diverse attachments: the main board or motherboard.
- To accommodate and manage the previously mentioned external ports and internal parts, the motherboard is loaded with hierarchical circuitry. It's like a mass of interconnected highways or conveyor belts. Signals travel in from the port or part, reach a hub, proceed to a later hub, and so forth. As a speedy rest stop for long-running work in progress, the temporary memory is a frequent start or end. The intricacy of contemporary device links ensure that motherboards are both busy and sophisticated, yet once more the overall task is unglamorous transportation. There's a further clue for continuing the search for thinking, though. For these transportation requests to be orderly and appropriate, the requests' source has to be the laptop's thinking. That source is the central processing unit (CPU).
- Analysis of the CPU risks a rapid slide into complexity and the specifics of individual models. At an abstract level, the CPU is divided into separate sections with designated roles. One is loading individual instructions for execution. Another is breaking down those instructions into elemental activities of actual CPU sections. A few out of many categories of these numerous elemental activities are rudimentary mathematical operations, comparisons, copying sets of bits (binary digits, either zeros or ones) among distinct areas in the CPU's working memory, rewriting which instruction is next, and dispatching sets of bits in and out of the CPU. In any case, the sections' productive cooperation consists of transporting bits from section to section at the proper times. Again setting aside mere transporting, the remaining hideout for the laptop's thinking is somewhere inside those specialized CPU sections completing the assigned elemental activities.
- Also considered at an abstract level, these CPU sections in turn are built from myriad tiny "gates": electronics organized to produce differing results depending on differing combinations of electricity flowing in. For example, an "AND" gate earns its name through emitting an "on" electric current when the gate's first entry point AND the second have "on" currents. Odd as it may sound, various gates ingeniously laid out, end to end and side by side, can perfectly perform the elemental activities of CPU sections. All that's demanded is that the information has consistent binary (bit) representations, which map directly onto the gates' notions of off and on. The elemental activities are performed on the information as the matching electric currents are transported through the gates. And since thinking is vastly more intriguing than dull transportation of information in any form, the hunt through the laptop needs to advance from gates to...um...er...uh...
This expedition was predictably doomed from the beginning. Peering deeper doesn't uncover a sharp break between "thinking" and conducting bits in complicated intersecting routes. No, the impression of thought is generated via algorithms, which are engineered arrangements of such routes. The spectacular whole isn't discredited by its unremarkable pieces. Valuable qualities can "emerge" from a cluster of pieces that don't have the quality in isolation. In fact, emergent qualities are ubiquitous, unmagical, and important. Singular carbon atoms don't reproduce, but carbon-based life does.
Ultimately, greater comprehension forces the recognition that the laptop's version of thinking is an emergent quality. Information processing isn't the accomplishment of a miraculous segment of it; it's more like the total collaborative effect of its abundant unremarkable segments. An outsider might scoff that "adding enough stupid things together yields something smart", but an insider grasps that the way those stupid things are "added" together makes a huge difference.
Readers can likely guess the conclusion: this understanding prepares someone to contemplate that all versions of thinking could be emergent qualities. Just as the paths in the laptop were the essence of its information processing, what if the paths in creatures' brains were the essence of their information processing? Laptops don't have a particular segment that supplies the "spark" of intelligence, so what if creatures' brains don't either? Admittedly, it's possible to escape by objecting that creatures' brains are, in some unspecified manner, fundamentally unlike everything else made of matter, but that exception seems suspiciously self-serving for a creature to propose...
Ultimately, greater comprehension forces the recognition that the laptop's version of thinking is an emergent quality. Information processing isn't the accomplishment of a miraculous segment of it; it's more like the total collaborative effect of its abundant unremarkable segments. An outsider might scoff that "adding enough stupid things together yields something smart", but an insider grasps that the way those stupid things are "added" together makes a huge difference.
Readers can likely guess the conclusion: this understanding prepares someone to contemplate that all versions of thinking could be emergent qualities. Just as the paths in the laptop were the essence of its information processing, what if the paths in creatures' brains were the essence of their information processing? Laptops don't have a particular segment that supplies the "spark" of intelligence, so what if creatures' brains don't either? Admittedly, it's possible to escape by objecting that creatures' brains are, in some unspecified manner, fundamentally unlike everything else made of matter, but that exception seems suspiciously self-serving for a creature to propose...
Saturday, May 02, 2015
data : code :: concept : verification
I've sometimes mused about whether my eventual embrace of a Pragmatism-esque philosophy was inevitable. The ever-present danger in musings like this is ordinary hindsight bias: concealing the actual complexity after the fact with simple, tempting connections between present and past. I can't plausibly propose that the same connections would impart equal force on everyone else. In general, I can't rashly declare that everyone who shares one set of similarities with me is obligated to share other sets of similarities. Hastily viewing everyone else through the tiny lens of myself is egocentrism, not well-founded extrapolation.
For example, I admit I can't claim that my career in software development played an instrumental role in the switch. I know too many competent colleagues whose beliefs clash with mine. At the same time, a far different past career hasn't stopped individuals in the Clergy Project from eventually reaching congenial beliefs. Nevertheless, I can try to explain how some aspects of my specific career acted as clues that prepared and nudged me. My accustomed thought patterns within the vocational context seeped into my thought patterns within other contexts.
During education and on the job, I encountered the inseparable ties between data and code. Most obviously, the final data was the purpose of running the code (in games the final data was for immediately synthesizing a gameplay experience). Almost as obvious, the code couldn't run without the data flowing into it. Superficially, in a single ideal program, code and data were easily distinguishable collaborators taking turns being perfect. Perhaps a data set went in, and a digest of statistical measurements came out, and the unseen code might have ran in a machine on the other side of the internet.
At a more detailed level of comprehension, and in messy and/or faulty projects cobbled together from several prior projects, that rosy view became less sensible. When final data was independently shown to be inaccurate, the initial cause was sometimes difficult to deduce. Along the bumpy journey to the rejected result, data flowed in and out of multiple avenues of code. Fortunately the result retained meaningfulness about the interwoven path of data and code that led to it, regardless of its regrettable lack of meaningfulness in regard to its intended purpose. It authentically represented a problem with that path. Thus its externally checked mistakenness didn't in the least reduce its value for pinpointing and resolving that path's problems.
That wasn't all. The reasoning applied to flawless final data as well, which achieved two kinds of meaningfulness. Its success gave it metaphorical meaningfulness in regard to satisfying the intended purpose. But it too had the same kind of meaningfulness as flawed final data: literal meaningfulness about the path that led to it. It was still the engineered aftereffect of a busy model built out of moving components of data and code—a model ultimately made of highly organized currents of electricity. It was a symbolic record of that model's craftsmanship. Its accurate metaphorical meaning didn't erase its concrete roots.
The next stage of broadening the understanding of models was to incorporate humans as components—exceedingly sophisticated and self-guiding components. They often introduced the starting data or reviewed the ultimate computations. On top of that, they were naturally able to handle the chaotic decisions and exceptions that would require a lot more effort to perform with brittle code. Of course the downside was that their improvisations could derail the data. Occasionally, the core of an error was a human operator's unnoticed carelessness filling in a pivotal element two steps ago. Or a human's assumptions for interpreting the data were inconsistent with the assumptions used to design the code they were operating.
In this sense, humans and code had analogous roles in the model. Each were involved in carrying out cooperative series of orderly procedures on source data and leaving discernible traces in the final data. The quality of the final data could be no better than the quality of the procedures (and the source data). A model this huge was more apt to have labels such as "business process" or "information system", abbreviated IS. Cumulatively, the procedures of the complete IS acted as elaborations, conversions, analyses, summations, etc. of the source data. Not only was the final data meaningful for inferring the procedures behind it, but the procedures in turn produced greater meaningfulness for the source data. Meanwhile, they were futilely empty, motionless, and untested without the presence of data.
Summing up, data and code/procedures were mutually meaningful throughout software development. As mystifying as computers appeared to the uninitiated, data didn't really materialize from nothing. Truth be told, if it ever did so, it would arouse well-justified suspicion about its degree of accuracy. "Where was this figure drawn from?" "Who knows, it was found lying on the doorstep one morning." Long and fruitful exposure to this generalization invited speculation of its limits. What if strict semantic linking between data and procedures weren't confined to the domain of IS concepts?
A possible counterpoint was repeating that these systems were useful but also deliberately limited and refined models of complex realities. Other domains of concepts were too dissimilar. Then...what were those unbridgeable differences, exactly? What were the majority of beneficial concepts, other than useful but also deliberately limited and refined models? What were the majority of the thoughts and actions to verify a concept, other than procedures to detect the characteristic signs of the alleged concept? What were the majority of lines of argument, other than abstract procedures ready to be reran? What were the majority of secondary cross-checks, other than alternative procedures for obtaining equivalent data? What were the majority of serious criticisms to a concept, other than criticisms of the procedures justifying it? What were the majority of definitions, other than procedures to position and orient a concept among other known concepts?
For all that, it wasn't that rare for these other domains to contain some lofty concepts that were said to be beyond question. These were the kind whose untouchable accuracy was said to spring from a source apart from every last form of human thought and activity. Translated into the IS perspective, these were demanding treatment like "constants" or "invariants": small, circular truisms in the style of "September is month 9" and "Clients have one bill per time period". In practice, some constants might need to change from time to time, but those changes weren't generated via the IS. These reliable factors/rules/regularities furnished a self-consistent base for predictable IS behavior.
Ergo, worthwhile constants never received and continually contributed. They were unaffected by data and procedures yet were extensively influential anyway. They probably had frequent, notable consequences elsewhere in the IS. Taken as a whole, those system consequences strongly hinted the constants at work—including tacit constants never recognized by the very makers of the system. Like following trails of breadcrumbs, with enough meticulous observation, the backward bond from the system consequences to the constants could be as certain as the backward bond from data to procedures.
In other words, on the minimal condition that the constants tangibly mattered to the data and procedures of the IS, they yielded accountable expectations for the outcomes and/or the running of the IS. The principle was more profound when it was reversed: total absence of accountable expectations suggested that the correlated constant itself was either absent or at most immaterial. It had no pertinence to the system. Designers wishing to conserve time and effort would be advised to ignore it altogether. It belonged in the routine category "out of system scope". By analogy, if a concept in a domain besides IS declined the usual methods to be reasonably verified, and distinctive effects of it weren't identifiable in the course of reasonably verifying anything else, then it corresponded to neither data nor constants. Its corresponding status was out of system scope; it didn't merit the cost of tracking or integrating it.
As already stated, the analogy wasn't undeniable nor unique. It didn't compel anyone with IS expertise to reapply it to miscellaneous domains, and expertise in numerous fields could lead to comparable analogies. There was a theoretical physical case for granting it wide relevance, though. If real things were made of matter (or closely interconnected to things made of matter), then real things could be sufficiently represented with sufficient quantities of the data describing that matter. If matter was sufficiently represented, including the matter around it, then the ensuing changes of the matter were describable with mathematical relationships and thereby calculable through the appropriate procedures. The domain of real things qualified as an IS...an immense IS of unmanageable depth which couldn't be fully modeled, much less duplicated, by a separate IS feasibly constructed by humans.
For example, I admit I can't claim that my career in software development played an instrumental role in the switch. I know too many competent colleagues whose beliefs clash with mine. At the same time, a far different past career hasn't stopped individuals in the Clergy Project from eventually reaching congenial beliefs. Nevertheless, I can try to explain how some aspects of my specific career acted as clues that prepared and nudged me. My accustomed thought patterns within the vocational context seeped into my thought patterns within other contexts.
During education and on the job, I encountered the inseparable ties between data and code. Most obviously, the final data was the purpose of running the code (in games the final data was for immediately synthesizing a gameplay experience). Almost as obvious, the code couldn't run without the data flowing into it. Superficially, in a single ideal program, code and data were easily distinguishable collaborators taking turns being perfect. Perhaps a data set went in, and a digest of statistical measurements came out, and the unseen code might have ran in a machine on the other side of the internet.
At a more detailed level of comprehension, and in messy and/or faulty projects cobbled together from several prior projects, that rosy view became less sensible. When final data was independently shown to be inaccurate, the initial cause was sometimes difficult to deduce. Along the bumpy journey to the rejected result, data flowed in and out of multiple avenues of code. Fortunately the result retained meaningfulness about the interwoven path of data and code that led to it, regardless of its regrettable lack of meaningfulness in regard to its intended purpose. It authentically represented a problem with that path. Thus its externally checked mistakenness didn't in the least reduce its value for pinpointing and resolving that path's problems.
That wasn't all. The reasoning applied to flawless final data as well, which achieved two kinds of meaningfulness. Its success gave it metaphorical meaningfulness in regard to satisfying the intended purpose. But it too had the same kind of meaningfulness as flawed final data: literal meaningfulness about the path that led to it. It was still the engineered aftereffect of a busy model built out of moving components of data and code—a model ultimately made of highly organized currents of electricity. It was a symbolic record of that model's craftsmanship. Its accurate metaphorical meaning didn't erase its concrete roots.
The next stage of broadening the understanding of models was to incorporate humans as components—exceedingly sophisticated and self-guiding components. They often introduced the starting data or reviewed the ultimate computations. On top of that, they were naturally able to handle the chaotic decisions and exceptions that would require a lot more effort to perform with brittle code. Of course the downside was that their improvisations could derail the data. Occasionally, the core of an error was a human operator's unnoticed carelessness filling in a pivotal element two steps ago. Or a human's assumptions for interpreting the data were inconsistent with the assumptions used to design the code they were operating.
In this sense, humans and code had analogous roles in the model. Each were involved in carrying out cooperative series of orderly procedures on source data and leaving discernible traces in the final data. The quality of the final data could be no better than the quality of the procedures (and the source data). A model this huge was more apt to have labels such as "business process" or "information system", abbreviated IS. Cumulatively, the procedures of the complete IS acted as elaborations, conversions, analyses, summations, etc. of the source data. Not only was the final data meaningful for inferring the procedures behind it, but the procedures in turn produced greater meaningfulness for the source data. Meanwhile, they were futilely empty, motionless, and untested without the presence of data.
Summing up, data and code/procedures were mutually meaningful throughout software development. As mystifying as computers appeared to the uninitiated, data didn't really materialize from nothing. Truth be told, if it ever did so, it would arouse well-justified suspicion about its degree of accuracy. "Where was this figure drawn from?" "Who knows, it was found lying on the doorstep one morning." Long and fruitful exposure to this generalization invited speculation of its limits. What if strict semantic linking between data and procedures weren't confined to the domain of IS concepts?
A possible counterpoint was repeating that these systems were useful but also deliberately limited and refined models of complex realities. Other domains of concepts were too dissimilar. Then...what were those unbridgeable differences, exactly? What were the majority of beneficial concepts, other than useful but also deliberately limited and refined models? What were the majority of the thoughts and actions to verify a concept, other than procedures to detect the characteristic signs of the alleged concept? What were the majority of lines of argument, other than abstract procedures ready to be reran? What were the majority of secondary cross-checks, other than alternative procedures for obtaining equivalent data? What were the majority of serious criticisms to a concept, other than criticisms of the procedures justifying it? What were the majority of definitions, other than procedures to position and orient a concept among other known concepts?
For all that, it wasn't that rare for these other domains to contain some lofty concepts that were said to be beyond question. These were the kind whose untouchable accuracy was said to spring from a source apart from every last form of human thought and activity. Translated into the IS perspective, these were demanding treatment like "constants" or "invariants": small, circular truisms in the style of "September is month 9" and "Clients have one bill per time period". In practice, some constants might need to change from time to time, but those changes weren't generated via the IS. These reliable factors/rules/regularities furnished a self-consistent base for predictable IS behavior.
Ergo, worthwhile constants never received and continually contributed. They were unaffected by data and procedures yet were extensively influential anyway. They probably had frequent, notable consequences elsewhere in the IS. Taken as a whole, those system consequences strongly hinted the constants at work—including tacit constants never recognized by the very makers of the system. Like following trails of breadcrumbs, with enough meticulous observation, the backward bond from the system consequences to the constants could be as certain as the backward bond from data to procedures.
In other words, on the minimal condition that the constants tangibly mattered to the data and procedures of the IS, they yielded accountable expectations for the outcomes and/or the running of the IS. The principle was more profound when it was reversed: total absence of accountable expectations suggested that the correlated constant itself was either absent or at most immaterial. It had no pertinence to the system. Designers wishing to conserve time and effort would be advised to ignore it altogether. It belonged in the routine category "out of system scope". By analogy, if a concept in a domain besides IS declined the usual methods to be reasonably verified, and distinctive effects of it weren't identifiable in the course of reasonably verifying anything else, then it corresponded to neither data nor constants. Its corresponding status was out of system scope; it didn't merit the cost of tracking or integrating it.
As already stated, the analogy wasn't undeniable nor unique. It didn't compel anyone with IS expertise to reapply it to miscellaneous domains, and expertise in numerous fields could lead to comparable analogies. There was a theoretical physical case for granting it wide relevance, though. If real things were made of matter (or closely interconnected to things made of matter), then real things could be sufficiently represented with sufficient quantities of the data describing that matter. If matter was sufficiently represented, including the matter around it, then the ensuing changes of the matter were describable with mathematical relationships and thereby calculable through the appropriate procedures. The domain of real things qualified as an IS...an immense IS of unmanageable depth which couldn't be fully modeled, much less duplicated, by a separate IS feasibly constructed by humans.
Thursday, November 03, 2011
to be agile is to adapt
Not too long ago, I read Adapt by Tim Harford. It's an engrossing presentation of a profound idea: beyond a particular bound of complexity, logical or top-down analysis and planning is inferior to creative or bottom-up variations and feedback. Adaptation can be indispensable. Often, humans don't know enough for other approaches to really work. They oversimplify, refuse to abandon failing plans, and force the unique aspects of "fluid" situations into obsolete or inapplicable generalizations. They're too eager to disregard the possible impact of "local" conditions. Biological evolution is the prime example of adaptation, but Harford effectively explores adaptation, or non-adaptation, in economies, armies, companies, environmental regulations, research funding, and more. Although the case studies benefit from adept narration, some go on for longer than I prefer.
Software developers have their own example. Adapting is the quintessence of Agile project management1. As explained in the book, adaptive solutions exploit 1. variation, 2. selection, and 3. survivability. Roughly speaking, variation is attempting differing answers, selection is evaluating and ranking the answers, and survivability is preventing wrong answers from inflicting fatal damage.
Agile projects have variation through refactoring and redesign while iterations proceed. Agile code is rewritten appropriately when the weaknesses of past implementations show up in real usage. Agile developers aren't "wedded" to their initial naive thoughts; they try and try again.
Agile projects have selection through frequent and raw user feedback. Unlike competing methodologies with excessive separation between developers and users, information flows freely. Directly expressed needs drive the direction of the software. The number of irrelevant or confusing features is reduced. Developers don't code whatever they wish or whatever they inaccurately guess about the users.
Agile projects have survivability through small and focused cycles. The software can't result in massive failure or waste because the cost and risk are broken up into manageable sections. Agile coaches repeat a refrain that resembles the book's statements: your analysis and design is probably at least a little bit wrong, so it's better to find out sooner and recover than to compound those inevitable flaws.
1Of course, the priority of people over process is also quintessential.
Software developers have their own example. Adapting is the quintessence of Agile project management1. As explained in the book, adaptive solutions exploit 1. variation, 2. selection, and 3. survivability. Roughly speaking, variation is attempting differing answers, selection is evaluating and ranking the answers, and survivability is preventing wrong answers from inflicting fatal damage.
Agile projects have variation through refactoring and redesign while iterations proceed. Agile code is rewritten appropriately when the weaknesses of past implementations show up in real usage. Agile developers aren't "wedded" to their initial naive thoughts; they try and try again.
Agile projects have selection through frequent and raw user feedback. Unlike competing methodologies with excessive separation between developers and users, information flows freely. Directly expressed needs drive the direction of the software. The number of irrelevant or confusing features is reduced. Developers don't code whatever they wish or whatever they inaccurately guess about the users.
Agile projects have survivability through small and focused cycles. The software can't result in massive failure or waste because the cost and risk are broken up into manageable sections. Agile coaches repeat a refrain that resembles the book's statements: your analysis and design is probably at least a little bit wrong, so it's better to find out sooner and recover than to compound those inevitable flaws.
1Of course, the priority of people over process is also quintessential.
Sunday, September 11, 2011
peeve no. 265 is users blaming the computer
No, user of the line-of-business program, the computer isn't the trouble-maker. It could be from time to time, if its parts are old or poorly-treated, but problems at that level tend to be much more noticeable than what you're describing. Generally, computers don't make occasional mistakes at random times. Despite what you may think, computers are dogged rather than smart. Computers do as instructed, and by "instructed" I mean nothing more than configuring the electricity to move through integrated circuits in a particular way. Computers can't reject or misunderstand instructions. No "inner presence" exists that could possibly do so.
I understand that placing blame on "the computer" can be a useful metaphor for our communication. But the distinction I'm drawing this time is substantive. To identify the precise cause of the issue that you've reported, a more complete picture is necessary. Your stated complaints about the computer's misdeeds really are complaints about something else. The reason to assign blame properly isn't to offer apologies or excuses. Figuring out the blame is the first step in correcting the issue and also in preventing similar issues.
I understand that placing blame on "the computer" can be a useful metaphor for our communication. But the distinction I'm drawing this time is substantive. To identify the precise cause of the issue that you've reported, a more complete picture is necessary. Your stated complaints about the computer's misdeeds really are complaints about something else. The reason to assign blame properly isn't to offer apologies or excuses. Figuring out the blame is the first step in correcting the issue and also in preventing similar issues.
- Possibility one is a faulty discussion of the needed behavior for the program, way back before any computer played a role. Maybe the right set of people weren't consulted. Maybe the right people were involved, but they forgot to mention many important details. Maybe the analyst missed asking the relevant questions. Now, since the program was built with this blind spot, the issue that you reported is the eventual result.
- Possibility two is a faulty translation of the needed behavior into ideas for the program. Maybe the analyst assumed too much instead of asking enough questions. Maybe the analyst underestimated the wide scope of one or more factors. Maybe the analyst was too reluctant to abandon an initial idea and overextended it. Maybe the analyst neglected to consider rare events that are not so rare.
- Possibility three is faulty writing of the program itself. Maybe the coders overestimated their understanding of their tools and their work. Maybe the coders had comprehensive knowledgeable and didn't correctly or fully express what they intended. Maybe a fix had unfortunate side effects. Maybe the tests weren't adequate.
- Possibility four is faulty data. Like blaming the computer, blaming the data is a symptom. Maybe something automated quit abruptly. Maybe manual entry was sloppy. Maybe the data is accurate and nevertheless unexpected. Maybe someone tried to force shortcuts. Maybe management is neither training nor enforcing quality control.
- Possibility five is faulty usability, which faulty data might accompany. "Usable" programs ease information processing from the standpoint of the user. Maybe the program isn't clear about what the user can do next. Maybe unknown terminology is everywhere. Maybe needless repetition encourages boredom and mistakes. Maybe, in the worst cases, staff decide to replace or supplement the program with pen marks on papers or fragile spreadsheets containing baroque formulae. Downfalls in usability may disconnect excellent users from excellent programs.
- Possibility six is the dreaded faulty organization, in which various units disagree or the decision-makers are ignorant. Maybe definitions are interpreted differently. Maybe the "innovators" are trying to push changes informally. Maybe the realm of each unit's authority are murky and negotiable at best. Maybe units are intentionally pulling in opposite directions. Regardless, the program probably will fail to reconcile the inherent contradictions across the organization.
Tuesday, August 23, 2011
git's index is more than a scratchpad for new commits
Someone relatively inexperienced with git could develop a mistaken impression about the index. After referring to the isolated commands on a quick-reference guide or on a "phrasebook" that shows git equivalents to other VCS commands, the learner might, with good reason, start to consider the index as a scratchpad for the next commit. The most common tasks are consistent with that concept.
However, this impression is limiting. More accurately viewed, the index is git's entire "current" view of the filesystem. Commits are just saved git views of the filesystem. Files that the user has added, removed, modified, renamed, etc. aren't included in git's view of the filesystem until the user says, with "git add" for example. With the exception of before the very first commit, the index is unlikely to ever be empty. It isn't truly a scratchpad, then. When checking out a commit, git changes its current view of the filesystem to match that commit; therefore it changes the index. Through checkouts, history can be used to populate git's view of the filesystem. Through adds, the actual filesystem can be used to populate git's view of the filesystem. Through commits, git's view of the filesystem can be stored for future reference as a descendant of the HEAD.
Without this understanding, usage of "git reset" is infamous for causing confusion. With it, the confusion is lessened. A reset command that changes the index, which happens in the default or with option --hard, is like a checkout in that it changes git's view to the passed commit. (Of course the reset also moves the branch ref and HEAD, i.e. the future parent of the next commit.) A reset command that doesn't change the index, which happens with option --soft, keeps git's "view" the same as if it remained at the old commit. A user who wanted to collapse all of the changes on a branch into a single commit could possibly checkout that branch, git reset --soft to the branch ancestor, and then commit. Depending on the desired effect, merge --squash or rebase --interactive might be more appropriate, though.
Post-Script: Since this is aimed at git newcomers, I should mention that before trying to be too fancy with resets, become close friends with "reflog" and "stash".
Post-Script The Second: Drat. The Pro Git blog addressed the same general topic, but based more directly around "reset". And with attractive pictures. And a great reference table at the bottom.
However, this impression is limiting. More accurately viewed, the index is git's entire "current" view of the filesystem. Commits are just saved git views of the filesystem. Files that the user has added, removed, modified, renamed, etc. aren't included in git's view of the filesystem until the user says, with "git add" for example. With the exception of before the very first commit, the index is unlikely to ever be empty. It isn't truly a scratchpad, then. When checking out a commit, git changes its current view of the filesystem to match that commit; therefore it changes the index. Through checkouts, history can be used to populate git's view of the filesystem. Through adds, the actual filesystem can be used to populate git's view of the filesystem. Through commits, git's view of the filesystem can be stored for future reference as a descendant of the HEAD.
Without this understanding, usage of "git reset" is infamous for causing confusion. With it, the confusion is lessened. A reset command that changes the index, which happens in the default or with option --hard, is like a checkout in that it changes git's view to the passed commit. (Of course the reset also moves the branch ref and HEAD, i.e. the future parent of the next commit.) A reset command that doesn't change the index, which happens with option --soft, keeps git's "view" the same as if it remained at the old commit. A user who wanted to collapse all of the changes on a branch into a single commit could possibly checkout that branch, git reset --soft to the branch ancestor, and then commit. Depending on the desired effect, merge --squash or rebase --interactive might be more appropriate, though.
Post-Script: Since this is aimed at git newcomers, I should mention that before trying to be too fancy with resets, become close friends with "reflog" and "stash".
Post-Script The Second: Drat. The Pro Git blog addressed the same general topic, but based more directly around "reset". And with attractive pictures. And a great reference table at the bottom.
Tuesday, August 16, 2011
why it isn't done yet
Modifying decrepit C code ain't like dusting crops, boy! Without precise calculations we could fly right through a star or bounce too close to a supernova, and that'd end your trip real quick, wouldn't it.
Saturday, August 13, 2011
the dash hole principle
The cigarette lighter receptacle has an amusing name. In my automobile and many others that I've seen, the present form isn't actually usable for lighting cigarettes. Now it's a round hole in the dashboard with a cover that's labeled as a power outlet. Over time, cigarette lighter receptacles turned into dash holes. The users of an object emphasized the secondary applications of it until the object itself dropped its primary application. It changed meaning through inventive usage.
Software users can be expected to act the same. Software developers should accept that the users, acting like humans, will adapt by introducing their own concepts and assumptions to a "finished" project. As DDD advises, the key is their language. When they speak about the software, and therefore the underlying design or data model, their words throw attention onto their interpretation of the "problem domain". They might describe data groups/categories and store their evolving understanding with rigid entries, like attaching "special" semantics to product identifiers that start with "Q". They might take several hours to run a series of automated preexisting reports, stuff the conglomerated figures into a spreadsheet, and then generate a chart - additional work which could all be accomplished by a computer in a tenth of the time.
The point is, software in the hands (and brains) of users can easily become a dash hole: an original design that came to be viewed much differently in practice. Developers who don't meet the needs of users will be bypassed manually as time goes on. In some cases, this may be a good approach. Some changes in usage just don't justify substantial software modifications. However, to state the obvious, not everyone is a good software analyst. Ad hoc solutions, enforced not by the software but by scores of unwritten rules, are prone to causing data duplication due to no normalization, chaos due to employee turnover or mere human frailty, and tediousness due to not thinking thoroughly about the whole process.
Dash holes function as adequate power outlets. But imagine if irritating dash holes could've been replaced with something designed to serve that purpose.
Software users can be expected to act the same. Software developers should accept that the users, acting like humans, will adapt by introducing their own concepts and assumptions to a "finished" project. As DDD advises, the key is their language. When they speak about the software, and therefore the underlying design or data model, their words throw attention onto their interpretation of the "problem domain". They might describe data groups/categories and store their evolving understanding with rigid entries, like attaching "special" semantics to product identifiers that start with "Q". They might take several hours to run a series of automated preexisting reports, stuff the conglomerated figures into a spreadsheet, and then generate a chart - additional work which could all be accomplished by a computer in a tenth of the time.
The point is, software in the hands (and brains) of users can easily become a dash hole: an original design that came to be viewed much differently in practice. Developers who don't meet the needs of users will be bypassed manually as time goes on. In some cases, this may be a good approach. Some changes in usage just don't justify substantial software modifications. However, to state the obvious, not everyone is a good software analyst. Ad hoc solutions, enforced not by the software but by scores of unwritten rules, are prone to causing data duplication due to no normalization, chaos due to employee turnover or mere human frailty, and tediousness due to not thinking thoroughly about the whole process.
Dash holes function as adequate power outlets. But imagine if irritating dash holes could've been replaced with something designed to serve that purpose.
Wednesday, March 23, 2011
cognitive load reduction
Debates about how to write better code (i.e. fewer bugs) revolve around increasing maintainability, but not too long ago I recognized a related and perhaps fundamental criterion: cognitive load reduction. The fewer disparate items that a developer must contemplate simultaneously, 1) the lower the chance that a mistake will slip in unnoticed, 2) the greater the amount of attention left for the details of the problem/domain rather than the twists and turns of the code. When code is confusing and demanding to comprehend, the cognitive load is greater, and therefore it's more difficult to write, trace, debug, and modify.
Awareness of impact on cognitive load should change the choices that someone makes. Sure, the first task is to produce code that meets the known requirements. Yet developers shouldn't then neglect the second task of refining the code until it's sensible. Code has two audiences, machine and human. This is a lens for perceiving the usual code debates.
Awareness of impact on cognitive load should change the choices that someone makes. Sure, the first task is to produce code that meets the known requirements. Yet developers shouldn't then neglect the second task of refining the code until it's sensible. Code has two audiences, machine and human. This is a lens for perceiving the usual code debates.
- Units of code organization with hard boundaries reduce cognitive load by freeing the reader from looking through many peripheral lines to trace execution.
- Good names reduce cognitive load by freeing the reader from inferring what a variable is for.
- An easier build process reduces cognitive load by freeing the builder from rehearsing and reciting a series of error-prone manual steps.
- Version control that meets the team's needs reduces cognitive load by freeing the team from devising complicated workarounds.
- Domain models that match the way that everyone thinks (according to common agreement) reduce cognitive load by freeing them from continual lossy translation of one another's statements.
- Frameworks reduce cognitive load by freeing the reader from examining custom-made immature solutions to ordinary incidental problems, e.g. templating, MVC, protocols. On the other hand, obtrusive frameworks may increase cognitive load by overshadowing and complexifying the base code without marginal benefit.
Sunday, March 20, 2011
calling a truce on sprocs
For a while, I've mostly been dismissive of database stored procedures or sprocs. The rationale is that databases are for storage ("Really, Capt. Obvious?"). By contrast, calculations, conditions, and data processing in general belong in a separate, dedicated tier; the clear benefit is a much more flexible, capable, reusable, and interoperable platform/language than the typical sproc. In this middle tier the intelligence resides in neatly divided objects that could potentially exploit different "persistence strategies" than the default database of choice. These objects presumably act as better models of the domain than collections of rows and columns. Application development happens on top of this middle tier rather than the database.
The opposite path is integration at the database level. Differing software all use the same "master" database. There may be a recurring import script that populates one or more tables with external data, entry interfaces that quite clearly manipulate rows and columns, canned reports whose queries become increasingly complicated. Knowledge of which tables to join or which column values to exclude spreads out through everything that performs a similar task. Analysts speak of the database as if it were the domain. Their first implementation question on new projects is "What tables do we need to add?"
Consequently, integration at the master database level can result in fragmentation and duplication. Enter sprocs. Essentially, a thoughtful agglomeration of limited and self-contained sprocs could take the place of a nonexistent middle/domain tier for some purposes. If everyone needs to run the same query all the time, at least putting it in a sproc will consolidate it. A complex calculation that everyone repeatedly makes could be computed in a single sproc. Ugly warts of the database model could have workarounds specified in sprocs.
Storage technology independence is lost with sprocs, but ongoing integration at the database level already makes that impossible. Sproc writing requires some learning but is offset by the considerable advantage of not having to rewrite the code in multiple clients. IDE support is less than ideal but a sproc shouldn't be too large anyway. Names and calls of sprocs are also rough but are likely to require less extra documentation than the alternative of laboriously touring table relations.
Sprocs: better than nothing.
Thursday, January 13, 2011
quick tip for using gitextensions with git svn
I imagine some others have already noticed this, but git svn commands can be added as "Scripts" that are available from the history context menu. For instance, for "git svn rebase": Go to Settings > tab Scripts, click Add, enter a Name and be sure to click "Add to revision grid context menu", enter "C:\Program Files\Git\bin\git.exe" for the Command, "svn rebase" for the Arguments, then click Save. Now, with a right-click on the history graph in gitextensions, the new command should show up down at the bottom with the entered Name. Choosing it will bring up the expected output window. Similar steps apply for "git svn dcommit --dry-run" and "git svn dcommit".
It's a small change, yes. And for the full glory of git you still must click on the little terminal icon on the gitextensions toolbar and use the command line. But repetitive tasks should be made as rapid and unobtrusive as possible to conserve the programmer's cognitive load (which is why running one's unit test suite should also be extremely easy and painless). Of course, even the tiniest enhancement to workflow adds up over many times thereafter. Just as a mostly-positive monthly cash flow is the key to long-term financial sustainability, a mostly-frictionless development flow allows programmers to expend their valuable time on stuff that matters, like design.
It's a small change, yes. And for the full glory of git you still must click on the little terminal icon on the gitextensions toolbar and use the command line. But repetitive tasks should be made as rapid and unobtrusive as possible to conserve the programmer's cognitive load (which is why running one's unit test suite should also be extremely easy and painless). Of course, even the tiniest enhancement to workflow adds up over many times thereafter. Just as a mostly-positive monthly cash flow is the key to long-term financial sustainability, a mostly-frictionless development flow allows programmers to expend their valuable time on stuff that matters, like design.
Tuesday, January 11, 2011
git is a VCS for the imperfect programmer
I just recently had a workday in which I realized how appropriate git can be for imperfect programmers. My department had released a project to the users for the first time, so I was working through the inevitable few tickets and/or change requests that ensue when software leaves a controlled development phase and collides with people.
I'd completed several commits on top of the release when I got a call about a bug related to a highly exceptional set of data. "No problem," I thought. "I can work it in with all these other commits and it will go out with them on the next scheduled full redeployment to the website." So I started working, but after designing a fix I discovered that it had so few systemic dependencies that I could easily push it out on its own without disruption, and enable the user who called to continue her work on the problematic data set. In another VCS this might have been unwieldy, but not with git. I stashed my work, created and checked out a branch at the last-released commit, popped the fix off the stash, and committed it.
Unfortunately it was only after that point that I noticed a possible weakness of the fix (and also the original code). Once again, with git this wasn't cause for alarm. I corrected the fix by amending the commit I'd just made. Finally, I had a file whose only difference from the last release was the fix. I deployed the file.
I should mention that an older edition of Subversion is the official VCS of my team, and we simply don't use branches or tags (it's not my decision). As a small team in a small organization with lots of informal communication, it's mostly sufficient for our needs, although I imagine that we'll need to become more sophisticated in the future. Thus, my fresh branch for the isolated bug-fix had to be a local git branch only. In order to ensure that the next deployment included that commit, I had to incorporate it into the Subversion trunk. With git, a rebase of my branch onto the HEAD of master was easy, and of course the actual merge of it into master was then a fast-forward. After a quick delete of the merged local branch and "git svn dcommit", everything matched up again.
A number of things about the procedure were imperfect. I'm an imperfect individual who made missteps. And I work in an imperfect setup with a decidedly imperfect team VCS. But it turns out that git fits these conditions just fine.
I'd completed several commits on top of the release when I got a call about a bug related to a highly exceptional set of data. "No problem," I thought. "I can work it in with all these other commits and it will go out with them on the next scheduled full redeployment to the website." So I started working, but after designing a fix I discovered that it had so few systemic dependencies that I could easily push it out on its own without disruption, and enable the user who called to continue her work on the problematic data set. In another VCS this might have been unwieldy, but not with git. I stashed my work, created and checked out a branch at the last-released commit, popped the fix off the stash, and committed it.
Unfortunately it was only after that point that I noticed a possible weakness of the fix (and also the original code). Once again, with git this wasn't cause for alarm. I corrected the fix by amending the commit I'd just made. Finally, I had a file whose only difference from the last release was the fix. I deployed the file.
I should mention that an older edition of Subversion is the official VCS of my team, and we simply don't use branches or tags (it's not my decision). As a small team in a small organization with lots of informal communication, it's mostly sufficient for our needs, although I imagine that we'll need to become more sophisticated in the future. Thus, my fresh branch for the isolated bug-fix had to be a local git branch only. In order to ensure that the next deployment included that commit, I had to incorporate it into the Subversion trunk. With git, a rebase of my branch onto the HEAD of master was easy, and of course the actual merge of it into master was then a fast-forward. After a quick delete of the merged local branch and "git svn dcommit", everything matched up again.
A number of things about the procedure were imperfect. I'm an imperfect individual who made missteps. And I work in an imperfect setup with a decidedly imperfect team VCS. But it turns out that git fits these conditions just fine.
Saturday, December 18, 2010
bow to the gitextensions cow
Recently I tried out gitextensions. A rhapsodic blog post seems to be in order.
There's an installer that includes msysgit and kdiff3. This means I haven't needed to download anything else to get started. The installer asked, up-front, how to handle the *nix/Windows line-ending issue and what to use for my name and email address. The GUI contains an easy way to edit .gitignore entries and it comes with default entries that are relevant to almost all Visual Studio development. It suggests and directly supports integration with the PuTTY tools for SSH authentication. This means I haven't needed to find and edit configuration files or go online to research recommended entries. As someone who considers himself to be at least minimally competent, I'm not phobic of manual configuration or command line usage, but why shouldn't the easy and predictable modifications be even easier?
My intense appreciation continued as I started using it. All the typical functions and their typical options are available. (Long-time git users doubtless prefer to perform the same tasks by rapid rote typing; there's an icon to pop open a "git bash" at any time, which is good to keep in mind.) Creating a branch is just a matter of entering a name when prompted, with a checkbox if you want to also immediately check it out.
The view includes the annotated history graph, the current working directory, and the current branch. Clicking on the branch name brings up a drop-down list of other branches. Choose one, and you check it out. Clicking on a commit in the graph brings up information about it in the bottom part of the screen, such as full commit details and the diff and the file hierarchy (each directory expandable and each file right-button-clickable for file-level commands like individual history). Clicking one commit then CTRL-clicking a second brings up the diff below.
Remember how git newbs tend to have trouble navigating the movements of files between the index and the working directory, especially before git became more friendly and talky? In gitextensions, the commit window simply has separate panes with buttons to move added/modified/deleted files in-between. There's also a button for amending. After the commit, or any other moderately-complicated operations, the git output pops up in a window for review.
Of course, pull, push, merge, rebase, cherry-pick, branch deletion are present, too. All are fairly straightforward assuming the user can follow the on-screen instructions and isn't completely ignorant about git. gitextensions has a manual that contains abundant screen captures, yet I imagine it's more useful as a reference for figuring out where/how in the GUI to accomplish a specific task than as a tutorial. I was pleasantly surprised by the smoothness of my first series of gitextensions conflict resolutions. kdiff3 came up, I chose the chunks and saved, then I clicked a continue button. Despite my later realization that I could've accomplished my goal through a more streamlined procedure, the end result was nevertheless perfect in the sense that I didn't need to apply a "fix-it" commit afterward (the credit likely should be split among git and kdiff3 and gitextensions).
My praise keeps going. gitextensions offers fine interfaces for "gc" and "recover lost objects", although thus far I haven't strictly needed either in my short usage span. It adds right-click items to the Windows file explorer. It adds both a toolbar and a menu to Visual Studio. If it isn't obvious, my personal preference is to keep the gitextensions GUI open all the time, supplemented by git-bash. On occasion, when I'm otherwise manipulating a file in explorer, I might invoke file operations right from there.
The remaining question is: are gitextension upgrades frictionless? Sooner or later the cow will tire of wearing that Santa hat...
Postlude: Farewell, Mercurial
Uh, this is uncomfortable. I'm sure you've heard this before, but it's not you, it's me. The cause definitely isn't something awful you did. You're still a great VCS that could make other developers very, very happy. I'm just looking for something else. My horizons have broadened a bit since we first met, and we don't think as alike as we did then. There are other options and considerations that I need to take into account. If I stuck with you forever, I worry that I'd become regretful or resentful. Some day, as we both change over time, I may come back to visit. Until then, I genuinely wish you well.
There's an installer that includes msysgit and kdiff3. This means I haven't needed to download anything else to get started. The installer asked, up-front, how to handle the *nix/Windows line-ending issue and what to use for my name and email address. The GUI contains an easy way to edit .gitignore entries and it comes with default entries that are relevant to almost all Visual Studio development. It suggests and directly supports integration with the PuTTY tools for SSH authentication. This means I haven't needed to find and edit configuration files or go online to research recommended entries. As someone who considers himself to be at least minimally competent, I'm not phobic of manual configuration or command line usage, but why shouldn't the easy and predictable modifications be even easier?
My intense appreciation continued as I started using it. All the typical functions and their typical options are available. (Long-time git users doubtless prefer to perform the same tasks by rapid rote typing; there's an icon to pop open a "git bash" at any time, which is good to keep in mind.) Creating a branch is just a matter of entering a name when prompted, with a checkbox if you want to also immediately check it out.
The view includes the annotated history graph, the current working directory, and the current branch. Clicking on the branch name brings up a drop-down list of other branches. Choose one, and you check it out. Clicking on a commit in the graph brings up information about it in the bottom part of the screen, such as full commit details and the diff and the file hierarchy (each directory expandable and each file right-button-clickable for file-level commands like individual history). Clicking one commit then CTRL-clicking a second brings up the diff below.
Remember how git newbs tend to have trouble navigating the movements of files between the index and the working directory, especially before git became more friendly and talky? In gitextensions, the commit window simply has separate panes with buttons to move added/modified/deleted files in-between. There's also a button for amending. After the commit, or any other moderately-complicated operations, the git output pops up in a window for review.
Of course, pull, push, merge, rebase, cherry-pick, branch deletion are present, too. All are fairly straightforward assuming the user can follow the on-screen instructions and isn't completely ignorant about git. gitextensions has a manual that contains abundant screen captures, yet I imagine it's more useful as a reference for figuring out where/how in the GUI to accomplish a specific task than as a tutorial. I was pleasantly surprised by the smoothness of my first series of gitextensions conflict resolutions. kdiff3 came up, I chose the chunks and saved, then I clicked a continue button. Despite my later realization that I could've accomplished my goal through a more streamlined procedure, the end result was nevertheless perfect in the sense that I didn't need to apply a "fix-it" commit afterward (the credit likely should be split among git and kdiff3 and gitextensions).
My praise keeps going. gitextensions offers fine interfaces for "gc" and "recover lost objects", although thus far I haven't strictly needed either in my short usage span. It adds right-click items to the Windows file explorer. It adds both a toolbar and a menu to Visual Studio. If it isn't obvious, my personal preference is to keep the gitextensions GUI open all the time, supplemented by git-bash. On occasion, when I'm otherwise manipulating a file in explorer, I might invoke file operations right from there.
The remaining question is: are gitextension upgrades frictionless? Sooner or later the cow will tire of wearing that Santa hat...
Postlude: Farewell, Mercurial
Uh, this is uncomfortable. I'm sure you've heard this before, but it's not you, it's me. The cause definitely isn't something awful you did. You're still a great VCS that could make other developers very, very happy. I'm just looking for something else. My horizons have broadened a bit since we first met, and we don't think as alike as we did then. There are other options and considerations that I need to take into account. If I stuck with you forever, I worry that I'd become regretful or resentful. Some day, as we both change over time, I may come back to visit. Until then, I genuinely wish you well.
Thursday, June 10, 2010
agile's effect on productivity
Someone needs to state the obvious for those who miss it. Agile software development is not a silver bullet for productivity. Agile is not something you "mix in" and yield the same result faster. Applying agile practices to the humongous project won't result in massive software at lightning speed. That's not the aim of agile.
Agile's main effect is a decrease in project latency, not an increase in throughput. Agile development means delivering minimal but user-confirmed business value sooner, rather than delivering maximal but doubtful business value once at the end. The hope is that by going to the trouble of more frequent releases, the software will grow gradually but surely, as actual usage not guessing motivates what the software needs to be. The economy of agile doesn't result in a greater quantity of software features per development time period, but it's certainly intended to result in a lesser number of wasteful software features per development time period.
This shift in perspective affects everybody's expectations, so agile development is more than a set of new habits for developers. Project managers need to break their world-changing ambitions into much smaller chunks. Users need to become more involved. Analysts need to laser-focus their requirement-gathering. Architects and modelers (may) need to recalibrate the balance between resource costs and the pursuit of perfection.
If a plane flight is like a software project, then agile development won't decrease the time it takes to go from Seattle to Boston non-stop. But it will change the non-stop flight into a series of short connecting flights, and by the time you reach Chicago you might realize that you didn't really need to go to Boston in the first place.
Agile's main effect is a decrease in project latency, not an increase in throughput. Agile development means delivering minimal but user-confirmed business value sooner, rather than delivering maximal but doubtful business value once at the end. The hope is that by going to the trouble of more frequent releases, the software will grow gradually but surely, as actual usage not guessing motivates what the software needs to be. The economy of agile doesn't result in a greater quantity of software features per development time period, but it's certainly intended to result in a lesser number of wasteful software features per development time period.
This shift in perspective affects everybody's expectations, so agile development is more than a set of new habits for developers. Project managers need to break their world-changing ambitions into much smaller chunks. Users need to become more involved. Analysts need to laser-focus their requirement-gathering. Architects and modelers (may) need to recalibrate the balance between resource costs and the pursuit of perfection.
If a plane flight is like a software project, then agile development won't decrease the time it takes to go from Seattle to Boston non-stop. But it will change the non-stop flight into a series of short connecting flights, and by the time you reach Chicago you might realize that you didn't really need to go to Boston in the first place.
Sunday, May 23, 2010
meaning through isomorphism
Moreover, Gödel's construction revealed in a crystal-clear way that the line between "direct" and "indirect" self-reference (indeed, between direct and indirect reference, and that's even more important!) is completely blurry, because his construction pinpoints the essential role played by isomorphism (another name for coding) in the establishment of reference and meaning. Gödel's work is, to me, the most beautiful possible demonstration of how meaning emerges from and only from isomorphism, and of how any notion of "direct" meaning (i.e., codeless meaning) is incoherent. In brief, it shows that semantics is an emergent quality of complex syntax, which harks back to my earlier remark in the Post Scriptum to Chapter 1, namely: "Content is fancy form."prelude and the problem
Months ago I finally finished reading Douglas Hofstadter's Metamagical Themas. Since it's a collection of columns about varied topics, I don't plan to comment on it with the same level of enthusiasm that I applied to I am a Strange Loop (if I were, I would've gotten around to it much sooner!). But one of its recurring ideas, also raised in Hofstadter's other books, is so fruitful that I can't resist rambling about it at excessive length: meaning through isomorphism. I'd further claim that its importance rivals and complements that of self-reference, which usually has the spotlight in commentary about Hofstadter's ideas.
The universal philosophical issue or "problem" of meaning is easily explained. It's undeniable that people experience meanings and that a meaning is a relation. One chunk of stuff "means" another chunk of stuff; noun _____ represents, defines, symbolizes, or analogizes noun _______. But how can people reconcile or harmonize this experience of meaning with a universe that, according to the best means of detection and reason, consists of pieces of matter whose interactions are consistently indifferent to relations of meaning? Does/Can meaning really exist? Assuming it does, then what is meaning, where is meaning, and how does meaning originate? I'll work my way back to this later.
isomorphisms
The preceding questions have many proposed answers. I'm convinced that Hofstadter's description of meaning through isomorphism is a pretty good one. A mathematical isomorphism has a rigorous logical definition, but in the looser sense intended here, an isomorphism is simply matching one or more parts of one aggregate with parts of another aggregate such that one or more relations between the parts of one aggregate remain valid between the matched parts in the other aggregate. In a word, relations in an isomorphism are "preserved". (In passing, note the circular definitions that an "aggregate" is a collection of "parts" and "parts" are anything in an "aggregate" collection.)
To take an elementary example, if one aggregate is the set of numbers (4, 8, 15) and the other aggregate is the set of numbers (16, 23, 42), then an isomorphism that preserves the relation "greater-than" could match 4 to 16 and 8 to 23 and 15 to 42 because 8 is greater than 4 and 23 is greater than 16, 15 is greater than 8 and 42 is greater than 23, etc. (Naturally, the relation of "subtraction" is NOT preserved since 8 - 4 = 4 and 23 - 16 = 7.)
At first glance, this may seem like dodging the question of meaning. Why should anyone care that, through a greater-than isomorphism, (4,8,15) "means" (16,23,42)? Well, that depends on the situation. Hypothetically, if someone's purpose involved the greater-than relation and he or she could more easily manipulate numbers less than 16, then that person could work on (4,8,15) and use the isomorphism to apply the results to (16,23,42). Imagine the depressing story of a pitiful calculating device that can only store its numbers with 4 bits each but its task is to find the maximum of 16, 23, 42. Still too trivial? Then ponder a comparable isomorphism between number sequences: taking the sequence 0..255, matching 0..127 to itself, and matching 128..255 to the sequence of -128..-1 . Now go read about the computer storage of signed integers to find out why this comparable isomorphism isn't a toy example at all.
Thus the basic idea is straightforward but its implications are surprisingly wide-ranging as shown by Hofstadter in his more mind-bending sections. His exemplar is the isomorphism in the incompleteness theorems between numbers and the symbols of a formal logic system, although he returns time and again to descriptions of the human capability for analogy, whether in the contexts of translation or recognition or self-image or creativity. A common thread is the logically-strange tendency to transcend by self-reference, which goes by labels like "quote", "break out", "go meta", "use-mention distinction".
applied to computers
However, apart from complicated self-reference, Hofstadter admits in his serious AI speculations that mere meaning through isomorphism remains simultaneously effortless for people yet bewildering to computer science. People can figure out an original isomorphism that works and then not rely on it beyond suitable limits, but a program can't. Whatever are the underlying mechanisms that originally evolved in the human brain for reaching the "correct" answer in a complex environment, no artificial program has quite the same ultra-flexible Frankenstein mix of (what appear from the outside to be) random walks, data mining, feedback, decentralization, simulation, and judgment. Given its mental acts, we shouldn't be shocked by the sheer quantity of lobes and connections and structure in the brain, and maybe its operation is less a monolithic program than an entire computer packed with programs interrupting one another to get a chance at directing the body's motor functions.
On the other hand, this conspicuous lack of an AI for pragmatic isomorphisms is all too familiar to application programmers like myself. Our job is to fill the gap by the routine imposition of meaning through isomorphism. That is, we try to create a combination of algorithms and data that's an isomorphism for a specific human problem, like running payroll for a company. In a similar fashion, the total computing system is a stack of increasingly-complicated isomorphisms of the application programmer's work. As Hofstadter writes in his columns on Lisp (and properly-educated programmers know), the top-level program is compiled into an isomorphic representation, then the next level down does its own compilation, and so forth, until the original program is in the executable form appropriate for the relevant non-virtual hardware. The towering edifice is an impressive illustration of the exponential power of isomorphisms to capture and translate meaning into, ultimately, an electrical flow.
(Some technically-minded readers may be questioning which "relations" are preserved by these isomorphisms. After all, one or more parts of the "pipeline" could possibly optimize the original code in any number of ways like function in-lining, variable removal, tail calls... In this case, the relations are abstract but are in the category "the effect of the original": order of statements that are dependent, values of constants including user-visible strings, access details of I/O performed. When relations like these in the original code aren't preserved, the pieces lower in the stack thereby fail to carry out actual isomorphisms that would express the meaning.)
applied to information theory, communications, art
Of course, meaning through isomorphism isn't the only theoretical framework around for understanding computer processing as a whole. The same claim could be made for information theory, which is throughly successful and in use every day. Fortunately, the two are compatible. Say that the communication channel's sender and receiver each have aggregates, and the message symbolically encoded and sent over the channel is an isomorphism between their aggregates. So then the symbols of the channel's message indicate each match from one aggregate's part to the other aggregate's part. Before the first symbol, the receiver is at maximum uncertainty or entropy about the isomorphism. After the first symbol and each symbol thereafter, the receiver can use knowledge of 1) the matches communicated previously, 2) its own aggregate, and 3) its own aggregate's relations between parts to make increasingly likely guesses about the remaining matches (or correct randomly sent errors on a noisy channel). In accordance with information theory, good "entropy coding" for this message would send no more bits to the receiver than are required in order for the receiver's knowledge of aggregates and relations to infer the rest of the isomorphism. The isomorphisms processed in a computer system allow for lower information entropy and therefore greater compression. The most interesting portions of a codec have the responsibility of using relations among parts of the media stream to trash or reconstruct some parts of the aggregate stream.
Given the compatibility between meaning through isomorphism and information theory, it's unsurprising that communication in general is perhaps its most natural manifestation. A language is an aggregate of spoken or written words. The universe is an aggregate of pure experience (I dare you to be more vague!). Hence a worded message is an isomorphism between language and the universe. Rather, the message is an aggregate of words that were purposefully selected in order to communicate, via the language-universe isomorphism, an aggregate of thoughts about pure experience. The relations preserved by this isomorphism are countless and endlessly varied. In the message "The quick brown fox jumps over the lazy dog", "brown" is an adjective to "fox" so the indicated fox must have the color brown in depictions, "jumps" is a verb to the subject "fox" so the indicated fox must be in the midst of a jumping action, "over" is a preposition connecting the phrase "The quick brown fox jumps" to the phrase "the lazy dog" so the indicated fox must be of higher elevation than the indicated dog. Part of the reason why computer parsing of raw human language is stymied is due to a computer's lack of a human's uncannily deep well of experiences to fuel a feedback loop between the comprehension of syntax and semantics. In practice, the nuanced syntax of sentence structure, word forms, and connectives nevertheless results in highly ambiguous statements that require worldly knowledge and/or context to disentangle.
How exactly people can decode their own words is a fiendish and glorious enigma that's convinced many speculators to tie language to the essence of humanity. Those who presume a soul frequently equate it to the explicitly verbal segments of intelligence (e.g. "confess with your lips"). It's definitely a truism that, of all earthly creatures, people have the most developed and subtle languages. They can organize their mental and social lives to levels of complexity that contrast with the simplicity of their materials. All their creations in language, art, and other domains can have symbolic depth.
applied to the distinction between form and content
These feats of abstractive composition and interpretation lead to a commonsense division between a work's "surface form" and its "meaningful content". For instance, a surface form of red is said to represent the inner content of the artist's aggression, and a milestone of any artistic genre is the point at which critics begin to tirelessly accuse its artists of betraying the genre's pioneers by producing "mindless copies" that mimic style (form) but without substance (content).
Finally I return to one of the points in the opening quote. The practical classification of an expression's qualities into form and content just described is contradicted by Hofstadter's pithy motto "content is fancy form". In his words, "[...]'content' is just a shorthand way of saying 'form as perceived by a very fancy apparatus capable of making complex and subtle distinctions and abstractions and connections to prior concepts' ". Elsewhere he writes that form and content are on a "continuum", and the gap between is "muddy". Form and content are simply the same expression evaluated on different strata or in different domains. Syntax and semantics are different in degree, not in kind. If someone isn't taken aback by this proposition then he or she might not grasp its import.
I've found it instructive to employ this perspective on John Searle's thought-provoking Chinese room argument. There's a person, completely ignorant of written Chinese, in a closed room armed with only the materials necessary to 1) receive a message in Chinese, 2) manually execute the algorithm of a program that can pass a Turing Test in Chinese, 3) send the reply generated back out of the room. Assume, as in any Turing Test, that the judge outside the room who sends the original message and receives the reply can only rely on the messages to determine whether the unknown correspondent knows Chinese. Since the algorithm in the Chinese room can pass the Turing Test by producing highly-convincing replies, isn't it true that 1) based on the replies the judge will conclude that the person in the Chinese room understands the messages and 2) the judge's conclusion is in direct contradiction to the person's actual level of knowledge of Chinese? If you grant these two points, then the Turing Test criterion for "understanding" fails to find the right answer for the person in the Chinese room. Now change the role of the Chinese room inhabitant from a person+algorithm into a computer program executing the same algorithm. Remember that the person in the Chinese room passed the Turing Test by "shuffling symbols" that had no meaning to him or her. Is it any more reasonable to think that a program that passes a Turing Test is doing anything more than "shuffling Chinese symbols" like the person in the Chinese room? The upshot of the argument is that no matter how well a message's form of linguistic symbols is processed, its content or meaning could still be unknown to the processor; understanding of form does not imply understanding of content so content cannot be form.
As I see it, the meaning-through-isomorphism interpretation leads to a disturbing viewpoint on the Chinese room argument (reminiscent of how the EPR "paradox" led to disturbing but theoretically-consistent results for quantum mechanics). I'm restricted to a single clue to deduce who in the Chinese room argument really knows the meaning/content/semantics of the message: the location of the aggregates to which the message's symbols are isomorphic. The argument postulates upfront that the person doesn't know the Chinese language. In other words, he or she doesn't have any information about the aggregate or relations of the Chinese language, but the message's symbols are parts of that unknown aggregate. Clearly no isomorphism happens in the person and so none of the intended meaning is there. I can agree that the person's lack of the necessary aggregate makes him or her clueless about the messages. But that leaves one possibility: the algorithm is the thing that understands the meaning. In the Chinese room, or indeed in any scenario alike to a Turing Test, in order for the algorithm to form convincing replies at all it must be able to decode the meaning, and to decode the meaning it must have the necessary aggregates to complete the isomorphism. Based on the usual opinion that understanding demonstrates intelligence, for the purposes of the Chinese room, the algorithm is more intelligent than the person. From the standpoint of the message's meaning, the person's participation in the communication is akin to functioning as the algorithm's tool, channel, or messenger (according the customary literary allusion, the algorithm is the person's "Cyrano de Bergerac"). When meaning occurs through isomorphism, there's no logical contradiction. The judge's guess that the person within knows Chinese is nothing more than an honest mistake. Don't blame the messenger for a passing Turing Test.
applied to philosophy and the brain
I called the idea of an intelligent algorithm "disturbing", but the delegation of various "intelligent" tasks isn't novel. Mathematical calculations were one of the first to be handed over to algorithms and devices. Then there's the long list of recommendations for sundry occurrences (do this when there's a fire, do that when someone needs resuscitation). And how much of the typical job is reducible to either rote actions or following new orders whose rationale is unknown?
The disquieting aspect of the imaginary Turing Test algorithm is the unprecedented usurping of the noblest of intellectual pursuits, understanding meaning. A traditional philosopher might declare that to be human is to understand and, furthermore, understanding is an accomplishment that can't be performed by anything else. Rocks don't understand. Plants don't understand. Animals don't understand but many are trainable. In contrast, humans experience a detailed "meaning mirror" of the universe that's symbolized in their languages. The "meaning mirror" has the name "mind" or "soul". Humans can understand the meaning of an expression by its effect on the "mind". In summary, meanings are ethereal inhabitants of minds, and only humans have minds.
Such a traditional explanation is appealing (to some people especially so) but it's complicated because it grants "first-class co-existence" to a purely mental/non-physical world. By making access to the cognitive world a special privilege of humans, it's dispiriting to the prospect of AI ever arising. It's also messy because people tend to embellish the details of the non-physical world in a multitude of opposing directions. It conflicts with the normal and productive method of the sciences, which is to find physical causes for phenomena. It outlines the existence of meanings but at numerous costs.
Dropping the traditional philosopher's explanation leaves the philosophical question of meaning unanswered...unless isomorphism is introduced in its place. Isomorphism furnishes a plausible intellectual underpinning for meaning in a solely materialistic universe. An isomorphism requires only materialistic ingredients for its elements: parts, aggregates, relations, matches.
What materials? The choices are everywhere, as boundless as creativity. Earlier, one set of materials was the software and hardware components of a computer system. In his writings Hofstadter has used DNA as a sterling example whose meaning is the proteins transcribed from it. And in keeping with the section on communication, any usable information channel is a candidate for meaning through isomorphism.
For philosophical concerns, the more relevant set of materials for isomorphism is the human brain. If we're to give up believing in our non-physical realities, we should reasonably expect a competitive brain-based theory of our mental capacities and qualia. I expect the brain's networks to be effective materials for isomorphisms. The combined excitatory and inhibitory network connections seem like prime building blocks for exquisite parts, aggregates, relations, and matches. Connections in general are implicitly essential to the definition of an isomorphism. An aggregate is parts that are connected, a relation is a connection between parts, a match is a connection between parts in separate aggregates.
One can then concede the, for lack of a better name, mind/matter isomorphism: the physical layout of the brain's network is directly responsible for all "non-physical" thoughts we feel. I don't suggest that there's a "grandmother neuron" but that the numerous neurons and even more numerous neuronal junctions, in response to the onset of the relevant stimuli, effect a mental experience of grandmother, whatever that may be. I neither suggest that one brain's network layout of isomorphisms resembles a second brain's except on a gross regional level; the variance among individuals in immediate word-association responses certainly makes a closer resemblance doubtful. I do suggest that, with sufficient prior knowledge about the isomorphism between the specific brain's network and its environment, a "scan" of the merely anatomical changes associated with the formation of a new long-term memory would enable the scanner to know with some certainty what the memory was "about". (I'm skeptical that anyone could figure out a workable procedure for it. Brain interfaces are getting better all the time, but the goal is to clumsily train the brain and the interface to work together, not to accurately read the brain's ephemeral signals.)
applied to objectivity
The proposed isomorphism between a person's brain and his or her encounters with reality puts not only the philosophical categories of "mind" and "matter" in a new light but also the categories of "objective" and "subjective" meaning. Objectively, whenever a scientist examines eroded canyons with sedimentary rock walls, maybe unearthing fossilized water-dwellers, he or she can assert the ancient history of the river that flowed there long ago. The river left traces so the river can be factually inferred. Also objectively, given the full network of a brain (and many secondary clues?), the brain's memories could be deduced in theory. The thoughts of the brain left traces so the thoughts can be factually inferred. At the time people would've called the brain's thoughts subjective, but with enough work the thoughts might as well be called objective.
Obviously the isomorphism in a human brain is incredibly dense and interwoven, which causes the complexity of the undertaking to have the magnitude of a perfectly accurate measurement and forecast of all the weather in North America. It's too hasty to proclaim it impossible, though. People have managed to translate hieroglyphics and break the Enigma code. The elusiveness of the "right" isomorphism doesn't disqualify it from discovery (well, not counting some exceptions like 1) the perfect elusiveness of "isomorphisms", like one-time pads, that by design have "matches" but preserve precisely zero relations, and 2) degradation/corruption of the aggregate's material medium).
Having seemingly decided that the derivation of meaning through isomorphism places it into the "objective" category, one could be forgiven for attempting to additionally place it into the "unambiguous and undebatable" category. In people's regular conversations, the two often go hand-in-hand; objective facts are brought in to silence the clamor of unfounded opinions. Not so for isomorphisms. In fact, an isomorphism's objective existence is why it can't have any inherent authority or precedence over other isomorphisms. Surely people can agree on any number and flavor of criteria for the selection of an isomorphism, but there's no physical coercion. (I could elaborate on how the personal selection of isomorphisms is supportive of pragmatic philosophy but I won't...)
x + y = 815. I'm writing a message to send in a bottle, but in what language? I see a person holding two fingers against his forehead and I take for granted that he has a headache. You mention a common acquaintance by the name "Alex" and I conclude you're speaking of "Alexander" not "Alexandra". "Ambiguity" is more or less shorthand for too many isomorphism candidates to pick.
In a much more formal context, computer scientists have established ever-growing sets of problems that are proven to be solvable by the same kind of algorithm. When a fast algorithm solves any problem in the set, it could attack the rest, too. On an algorithmic basis, the problems are isomorphic. A computer scientist searching for a generalized solution to the set of problems doesn't need to "prefer" one to the rest. (He or she need not be too depressed. In most applications a "good enough" or "partial" solution is adequate.)
The "relativity" (non-preference) of objective isomorphisms is mind-blowing to me. It turns the world "inside-out". The Earth is not the center of the physical universe. Neither am I the center of the universe of meaning. After a thunderclap, the impact sets air in motion. The sound wave is one set of molecules jiggling, then the next, then the next. By moving similarly, i.e. isomorphically, aren't the air molecules transmitting the "meaning" of the thunderclap? Eventually, the air in my ear canals moves after being pushed in turn. The movement of the drum corresponds to an isomorphic shift in electrical impulses (yeah, I know I'm simplifying it). The nerve cells isomorphically react and in so doing continue to pass along the "meaning" of the thunderclap into the brain that I like to refer to as "mine". The connections in that brain isomorphically mirror my stored memories, spilling over into my language centers. "I hear thunder." In the relative terms of all these isomorphisms, who's to argue that I'm the origin of my spoken thunder message? But I may not be the terminus, either. What if my statement motivates the people around me to leap into action? Aren't their actions isomorphic to my statement? My message has made their actions meaningful. It started with a thunderclap.
Monday, June 29, 2009
DVCS and good merges
While it may be true that a DVCS (i.e. multiple repositories) implies good support for branching and merging else the DVCS would be a nightmare to use, it's incorrect to logically conclude that good support for branching and merging implies that someone must be using a DVCS (A -> B does not mean B -> A). Decentralization or support for multiple repositories is a different question than whether the version control system has "dumb" merging. The benefits of easier collaboration (we can pull and push work-in-progress to each other without committing to the "canonical" repo) and offline work are more honest justifications for DVCS than mere good merges.
Thursday, September 04, 2008
the Algorithm will be incorrect
...and we the users (or maintenance programmers) write the pertinent definition of "incorrect". Software and hardware might execute flawlessly the exact steps proscribed by a set of exhaustive written policies. If that is the case, good jorb and a round of gold star stickers to everyone responsible. Still, the algorithm is incorrect each time it produces a different result than we think it should.
The way to cope is to plan ahead for inevitable incorrectness. As much as is reasonable, assumptions should be flexible. Options should be open-ended. Overrides should be available; users should be advised and protected from mistakes, but some users have their own good reasons to request something "incorrect".
A perfect algorithm will be incorrect. More poetically, a perfect algorithm is too good for this imperfect world.
The way to cope is to plan ahead for inevitable incorrectness. As much as is reasonable, assumptions should be flexible. Options should be open-ended. Overrides should be available; users should be advised and protected from mistakes, but some users have their own good reasons to request something "incorrect".
A perfect algorithm will be incorrect. More poetically, a perfect algorithm is too good for this imperfect world.
Thursday, June 19, 2008
if you want to be taken seriously...
...don't use Comic Sans MS in your email sig. Just don't. I suppose you can use it for your own name to try to give the impression that you're "just a simple, ordinary guy", but under no circumstances use it for your job title or employer. Form reinforces or contradicts content, people. Seeing the words "analyst", "engineer", "senior" in Comic Sans MS font is irrepressibly funny, and I will proceed to mentally point and laugh at your correspondence.
It's also rather disheartening to read the word "services" in Comic Sans MS as part of your employer's name. It really doesn't shore up your customers' confidence that they're buying high-quality expertise...
It's also rather disheartening to read the word "services" in Comic Sans MS as part of your employer's name. It really doesn't shore up your customers' confidence that they're buying high-quality expertise...
Subscribe to:
Posts (Atom)