Category Archives: coding

xbox 360 controllers + wireless dongle + mac

Peterson and I are here at the dinner table with my Microsoft wireless controller dongle and two wireless xbox360 controllers.

http://www.maclife.com/article/howtos/how_use_xbox_360_wireless_controller_your_mac led us to http://tattiebogle.net/index.php/ProjectRoot/Xbox360Controller/OsxDriver … and the source is available! It is made up of … a lot of xcode projects.

I’d used the tattiebogle.net driver before, on the day I saw that Microsoft was making the dongle again (they stopped making it for a few years, and Amazon was rife with fakes) when I was at Best Buy looking for something else, and had to get it. I played with it and ControllerMate a little bit that day, but didn’t get ControllerMate to successfully map to any keys.

We ran the tattiebogle driver again on Peter’s computer tonight, and one of the things the driver does is activate the rumble packs when you depress the triggers. We noticed that one side’s rumbling is much stronger than the other, so I asked the internets for information. http://www.ifixit.com/Guide/Installing-Xbox-360-Wireless-Controller-Vibration-Motors/3292/1 is like looking at porn! It does say that “The right motor has significantly more counter-weight than the left motor.”

Then I remembered that a few weeks ago Brycolyn gave us some guitar hero peripherals they’ve had lying around their house from a former roommate (they don’t have an xbox360 anymore). We got them out of the trunk.

Then there weren’t any AA batteries around, so we went to Lucky to get a whole bunch. I am now 48 Sunny Select brand AA batteries richer (plus some almond milk, ling ling potstickers, some paper towels, and some orange juice).

The guitars connect to the dongle just fine, and we’re messing around with the buttons. A, B, X and Y are what you expect (and what you know if you’ve navigated menus in rock band or played regular xbox games with your guitar as a controller). The interesting part is that it SEEMS that there’s an accelerometer or a gyroscope or something in the guitar–when you tilt it up, what the driver thinks is the right-side joystick changes position in accordance to the attitude of the device. When you shake it around, you can see it moving. When you make a CHANGE in orientation, what the driver thinks is the right trigger gets depressed, and then when you stop moving it, it goes back down to a slightly jittery middle phase. Must be how it knows about star power when you tilt it up. Requires more investigation.

I am also super tired. We went to Tahoe this weekend.

Hacking Health

I’m at the UC Berkeley Hacking Health hackathon! Friday-Saturday. Pitches are at 6pm tonight.

Fritz and I are making an iOS app to help parents log seizures. I would also like it to log all kinds of events: medication (and it’ll alert you when it’s time to take your next one), eating, pooping, sleeping, developmental stories…

Last night, BodyMedia and Mashery were here doing a demo of the BodyMedia API, and giving out devices to developers. They’re offering a prize for a hack that uses the device! It’s hilarious because I just started wearing a BodyMedia device last week and had no idea they’d be here. CONNECTIONS

We’re at the top of the Chase building, which I’ve walked by many, many, many times but have never been in. Gorgeous views all around. There’s a UC Berkeley startup incubator up here, which they tell me is pretty new.

I want to show you all my pictures of our ideation (lol that word) whiteboard, and of the view from the Skydeck, but I need a better way to get photos off my phone and onto WordPress. (Hi maiki, android, and owncloud!) Right now I either email my photos (can only do them individually) which sucks, or I wait for the dropbox import (I have to plug my phone in to my computer and wait, and wait, and wait) … for some reason my iPhone doesn’t show up as a hard drive on either my PC or Mac so I can’t just grab the photos. I could load up PhotoViewer or GrabPhoto or the thing that isn’t iPhoto on my Mac and wait and wait and wait for it to sync, and then scroll all the way to the bottom, and then select the ones I want to import, and then wait for the import, but I use the Dropbox feature because it automatically imports just the ones that haven’t been imported yet without me having to look through it. Garrrr it shouldn’t be this hard

The table next to us is doing a kinect hack! This makes me really happy! There’s also another group I’ve spoken to examining asthma and comorbidity. DATA SYNTHESIS IS EVERYWHERE

MonoRail: Monolithic Rails Application – Jack Danger – GoGaRuCo ’12

MonoRail:
@jackdanger from Square, Inc
1. rails applications go in a particular direction and you
2. problems primarily about ownership.

imagine you have not a monorail, but a green fields applications, you got featured on techcrunch, everything’s great.
fast forward 3 years in the future. you have a rails
several $30,000 mysql box.
over 1,000 controllers. but a lot of rails apps now are well in excess of that and who knows how many
spotty test coverage. no idea how much because rcov would take 7 years or segfault
sad developers. 22 devs talking about how they don’t want to walk on a big company anymore (cuz you have 30 employees) and they want to work on something where they can make a difference (laughs)
ultra-slow test suite

you might think, you can avoid this, this won’t happen to you. twitter, square, groupon, yellowpages, livingsocial, modcloth — all of these and more survived a monorail. they’ve pulled pieces of rails 2 code on 1.8.2 out and onto services that are manageable.
but look at the first day
“can rails scale?” is not a question. heroku ps up, tada, you’ve scaled rails (lol)
real questions: scaling your data, codebase, customers, feature count
codebase: git and github fixes this.
customers: communications, engaging with customers, and logging (so you can explain what happened
feature count: rails is actually great at this. you want a new feature? new path, new controller action, new database table, bam.

what we don’t talk about is scaling your developer headcount. there’s no pattern or tool that helps us do this.

in the beginning. this is you, on day one, with a green-field project. 1 app == 1 product == 1 developer. welcome aboard, put your code here — the one place your code is. twice the pace because you have 2 people! great pattern for early days.
want an admin interface? add /admin ! how about analytics? put /trends ! (though let’s get real — you’ll just put it into your view in the admin interface)
“rails can do anything!” “let’s make it do everything!”
this is the reason we get monorails is because let’s keep going this direction! little by little.
you can subclass actionmailer because you’re doing it only once. not good software design, but it works.
add email, phone, etc… validates_presence_of all of them… so easy to just add a set of methods inside of user.rb but it’s all right there, easiest way to get your feature out the door, testing if product should exist, this is where to put it.

nice mini-framework. throw that in ./lib ! also tests in ./spec/lib !”

problem: rails is optimized towards the beginning of the experience. (postgres great for most things lol).

examples of directions rails could go from the beginning. if you’ve worked on giant enterprise problems, they’d pick the “mature” side, but

young: 1 database. mature: many databases.
even in a single
select records from 1 table, grab those ids, go to another table with the ids in the query, grab records, does 3rd query, then present them to you . eager loading multiple queries. if you’re hitting multiple times anyway, why not look up user’s locations in riak or something? just can’t

young: MySQL or Postgres at 10,000 = you’re fine. but mature: postgres does things really well. postgres does not need to do an entire table copy and rewrite to change a default (mysql you would)

young: ActionMailer. mature: anything else. you should have an actionmailer project (on sinatra or something) and a rails project that tries to send mail. you have two application.html templates, one is for web, one is for email. those should be different jobs. done in a different project where they can do their own deploys (like for a copy change, they won’t have to deploy your

young: lots of data in ‘users’. mature: only authentication in ‘users’ table. that’s it! anything else is a feature that relates to that user. if you grow infinitely, it will be in a different

young: features sitting in ./lib. mature: internal gems. if in lib, needs to be extracted and put into a private gem. nothing should be there for more than 10 days or so. (and if you have object_patch.rb, just delete that)

young: validates_*_of. mature: database validations, else you’ll have corrupt data. you’ll have sad users.

young: default logging. rails will log what it does, but your app should log what it does too. mature: log every significant action.

young: analyze your data … in the main db. have sql do multi table joins, time series. problem: that data is in the wrong shape (in online transactional processing instead of by day), also interfering with . mature: analyze your data elsewhere. ETL – extract, transform, and load. (xavier) set up stuff with replicas, and star schemas (?), group things by interesting trends.
story: hitting reload took us down. “i would call this a weak point in your application”… fixed!

conway’s law.
the shape of your people shapes your app.

deterministic: if you put in the work, you have control. problem comes in when you have other people working on unrelated things, you could have a situation where your thing is great but doesn’t work. so instead, have each person or team work on a specific different thing. how to do this?

rails can do this, but it’ll take foresight.

companies that have a monorail: it starts when you decide to have teams

1. service interfaces. you need to build internal apis. not for everything, but for all the things that you think maybe someday shouldn’t live here anymore. like email. give bare minimum information about what you need for this task, with nothing about how it needs to be done. some team will say “we’ll take the other side of this service and fix it.” nothing you do will ever change this. no longer your concern because there’s a line between your responsibility and not your responsbility.

another example that’s made-up: marketing front pages (a signed-out experience), a signed-in experience using html, and an api. clear definitions of different sites, but all in same rails.
Marketing.render request, :with_condition => “public”
something can happen inside of there, it’s an omega mess, but its’ someone else’s responsbility.

2. extract the code
copy the app via git clone and delete the parts you don’t need on the other app, or you can rebuild the piece that you want from scratch. will have to modify a lot of files to do stuff like this. but you should be deleting or moving or adding files, not just modifying files.

3. move the data
across the databases. everybody’s got an interesting solution.

most important thing: recognize there’s a real human cost for not recogniz
as soon as we change our group structure, we’ll be in trouble. instead of building things we imagine, we’ll be maintaining someone else’s code.

we deserve to go into work every day and be happy.
ruby was optimized for happiness by matz, rails opt for happiness by dhh, mega rails not doing a good job of making us very happy.

RubyMotion – Amit Kumar – GoGaRuCo ’12

rubyist
consultant:tata consultancy services ltd
github: toamitkumar
twitter: toamit
toamikumar.github.com

1 man army. amit tried it out, liekd it, built 3 apps, 1 will be on app store soon
then 2 man army!

what exactly is rubymotion?
a commercial fork of macruby, a ruby implementation of mac osx.
it compiles into optimized machine code. assembly language by llvm.
automatic object memory allocation and reclaim. first moment of happiness!
compiles the intervaces built in IB of XCode – second moment of happiness
ruby-runtime tightly integrated with Obj-C runtime. Same ancestor as Obj-C.

Example:

you can share objects between your ruby code and your objc code with no performance cost. compiles into executables

rubymotion and objective c has a shared runtime.
has a shared ancestory, the foundation framework (NSObject – Kernel)
can call all sdk apis from your ruby applications.

Can I code using RubyMotion without the pain of learning the Cocoa Framework? (asked us to guess) No.
You want to control the code you’re writing. You don’t believe in magic. I want to build my interface myself with code.

Second question: when there are frameworks like PhoneGap, why should I care about RubyMotion?
Answer: pros and cons of phonegap. pros: easy to use, we all know html/js/css (“javascript is the next big thing”, chuckles). it’s a bridge. whenever you have a bridge, you have a performance bottleneck. also, apple doesn’t optimize uiwebview thread. so the cons: bridge that makes native api calls, runs in single thread of UIWebView which is slow, limited support of direct access to native apis, debugging becomes extremeley difficult when you downlo

how do i download rubymotion? $200
sites.fastspring.com/hipbyte/product/rubymotion
“i never regretted it”
after you download and install, you see the motion command.
$ motion –help
shows you motion create, activate, update, support. very cool because from your console, you have access to the suport ticket system of rubymotion. opens the default browser on your machine with certain fields filled out already: lic key, info about machine, you fill in the bug report or feature request.

if you want to stick to a certain version, you’ll have to use this
sudo motion update –force-version=1.2

rakefile
app delegate class
specs

let’s look at the motion rake command
in rubymotion, your bread and butter is rake. develop, test, and deploy: you’ll be using rake.
rake default: build the project, run the simulator.
rake spec: run the test/spec suite, comes with a rspec-like frameowrk called macbacon (?), all the flavors of rspec
rake build: builds the application
rake archive: creates an ipa file you can distribute or push to app store
build, test deploy. (those are the

rake config: every ios application has a bunch of configuration options.
name
delegate_class
frameworks (bunch of frameworks in cocoa. defaults: UIKit, Foundation, coreGraphics
device_family: iphone, ipad, or both

when you fire rake, starts compiles your ruby source code.
pro-tip: other configuration options
app.file_dependencies ‘appcontrollers/curves_controller.rb’=>’app/controllers/main_controller.rb’
app.vendor_project

let’s look at the soul of a rubymotion app.
console REPL: read evaluate print loop. demo!

$ rake
opens the simulator!! also builds project.
(main)>
is the prompt.
(main)> @controller = App.delegate.instance_variable_get(:@ui_view_controller)
=> #
(main)> @controller.view.backgroundColor = UIColor.whiteColor
=> UIColor.whiteColor(0.0)
wow, it makes the color white!
(main)> @segment = UISegmentedControl.bar([“Hello”, “Patient Attendees”])

makes it, but hasn’t addd to the view, so here
(main)> @controller.view <
it appeared! two blue buttons on the top left.

now move it 100px down and 200 px to the right. will use another RubyMotion gem, called sugarcube

include SugarCube::Adjust
adjust @controller.view.subviews[0]
d 200
(moves it down 200)
r 100
(moves it right 100)

this is one way of playing with it. there is another way. hold the command key, and move the mouse. hovering over UIView, then the prompt changes. then you click it, main has changed to the object that you have highlighted. clicked on

self.text
=> “Hello”
self.text =
quit (to get back to main)
(main)> tree
shows you hierarchy

you feel in-control. another moment of happiness (our fourth!)

an excellent in-browser demo of repl: pieceable.com/rubymotion-console

compiling
linking
packaging
code signing

testing
like rails, rubymotion comes bundled with an rspec-like framework called: MacBacon
a test that’s the starting point of your application. saying that more tests can be added to your application.
we can look at the test: has one window.
rake spec
launches simulator again, because it runs the test case in that environment.
passed!

MacBacon has almost all of the syntactic sugar that we are accustomed to in RSpec: assertions, matchers, before/after blocks.
view testing: loading the nib/xib/storyboard file.

Since we could not push to testflight, we set up continuous integration.
dev –> github –> webhook to jenkins server on a mac mini (important aspect to see here is the push from jenkins server to an app store we have ). over the air technology from apple, plist. browse to “app store” and see install button, then use a device to do that and test it.

can do these things:

rubygems – normal rubygems won’t work because rubymotion is statically compiled. rubymotion gems have to extend the configuration file. the authors of rubymotion gems have make sure compiler gets them.
uses bundler
lots of contributions from the commuity. big list. bubblewrap, teacup (stylesheet, moves away from building interface from code). sugarcube, formotion, some that he created and contributed back.
statically compiled :static
basic types in C have ruby counterparts, but for complex types in C, uses bridge

objective-c project
native-c
CocoaPods

RM is only ~4 months old, has a long way to go.
some things that it lacks:
– debugger (REPL kind of makes it easy)
– some dynamic code doesn’t work (“but hey, who cares?” lol)

some apps in the app store made by rubymotion:
everclip, cabify, survey (his app!)

15% discount off rubymotion if you email toamitkumar@gmail.com !

Grasping Complexity – Glenn Vanderburg – GoGaRuCo ’12

livingSocial
@glv

As programmers, we love simplicity.
harder to write/design, but pays off in the end.

Our world is binary.
because of the things we do. answers are true or false, right or wrong, the solution works or is broken. these are concepts we work with in code. that’s a good thing. but it carries over to other parts of our work. dealing with vague things is unsettling.

complete solutions.
we want to cover every edge case, complete solution. holy grail is both simple and complete. handles special cases without special
daniela told us a mathemetician’s joke yesterday, so today i’ll tell you a programmer’s joke.
zoologist says “omg a spotted zebra! no one else has disco
physicist says “hold on. at this point, all we know is that only one zebra
mathemetician says “sloppy thinkers. all this tells us is at least one side
programmer is saying “oh no. a special case!”

lol!
when i first heard that joke, thought it was programmers who aren’t very good. should be: “aha! spots are generalizations of stripes.” this turns out to be true. if you’re writing a shader, you can tweak it to make spots like a cheetah, or a tiger, or even stripes like a zebra. lol.

we like challenges. that’s why we like scala (lol) and obfuscated coding contests. we had one of these in living social.
here’s my solution: a yml parser that only uses regular expressions. haha. we like complexity sometimes.

but we don’t like complexity when it seems that the complexity has no bounds to it. there’s no way of knowing or pinning down all the little concepts we have to take care of.
sometimes, even the best solutions we can think of have downsides. we tend to recoil from these complex problems. if we are not able to completely run away and just ignore them, often, unfortunately, we pretend. we oversimplify, we act as if the problem were simple.
a motivation for where talk comes from… it’s election year. in our sound-byte and anger-driving political culture, we oversimplify.

will talk about: the ways we oversimplify, and jump to decisions (and make poor decisions) because we don’t face the full complexity of the problem ahead of us. then talk about some techniques/habits of mind we can cultivate to have us not recoil from complex problems, reach more sane solution for compromise in the face of complex problems we face as programmers.

one way we oversimplify:
we concentrate on primary effects, and ignore second-order effects.
story: business partners, one was top patent holder at dell. made process of getting patents through streamlined and cheaper, thought, we can build a . at first, thought: the world doesn’t need more patents. second pushback: it’s not a mechanical process, it’s a human process — the patent office is comprised of human officers; if there’s a product that funnels more patents into the process, they’l change the process. that’s an example of ignoring second-order effects and assuming the system you’re trying to change is not going to push back on you.

other way we oversimplify: forgetting secondary benefits.
when people teach tdd, they focus on the one benefit that’s the most interesting. if you only choose one benefit, it’s not a slam dunk… you have to remember all of them put together for a slam dunk. “wel there’s cheaper ways to catch errors” — but what about the other benefits?

asking the wrong question
sandi, in her talk yesterday, had a great example: pepole ask, what object should know this? perilous question because it assumes that one of the objects should. the answer “well, one that hasn’t been written yet” is not likely to occur to us. so instead, ask the question “what message should i send?” then write the code that way, and it might be clear that you need to write a new class to send that message.
story: why would the ops team do this? turns out th
story: why isn’t programming more like engineering? may be the wrong question. ask, which kind of engineering is it like?

binary thinking.
fall prey to the “sucks!” “rocks!” dichotomy.
story: yehuda: “v8 has a bug” – sometimes people dismiss things as soon as we find a flaw.
daniela touched on this: some say tdd doesn’t prove your app is perfect, but that doesn’t mean don’t use it

taking rhetoric at face value.
12 rules of xp, “you have to do all of them or you can’t call it that!” — but maybe you could take a few concepts and apply them and get a benefit.

ignore context.

giving up.
“some people are just good designers” — it’s magic.
“he just doesn’t get it.” what that tells me is: you haven’t taken the time to understand his objections.
“it just feels wrong” or “so beautiful and elegant” — what do those things mean?

what i’m trying to say is: there are useful techniques for grasping complexity, even when we have low-quality evidence for our solutions.

incomplete solutions
we should learn to seek these.
story: businessman who has built career out of tackling problems his engineers . 80% solution for 20% of the effort. 3 simple rules. for weight loss: never eat white foods. never eat in front of the tv. always park in the most distant parking space. these rules are not optimal. might not be the fastest way to lose weight. cauliflower is not bad for you at all. but keeping the rules simple means that people can remember them and stick to them; works with the way people think. seek heuristic or probabilistic solutions. heuristic reasoning used to get to approximate answer.
another example: git depends on a sha has for a bit of text. we KNOW that there CAN be mlutiple pieces of text with the same sha. it’s just so unlikely that it’s worth trusting that it won’t happen. has called github a globally distributed attack on sha1. (laughter) a precursor of that, the first thing i encountered of “yeah bad things can happen but we hope they won’t happen” – network file server in plan9 called venti. block-addressable file server, list of block addresses. the block address was the sha of its content. a lot of files change is small ways, so tomorrow’s version of the file might share all the same blocks instead of one at the end of the file. could have different versions. “the infinite improbability disk drive” haha
so we should think about how much liklihood of failure we can accept.

exploit power law distributions.
basic statistics: normal distributions, bell curves; doesn’t give us a lot to work with.

think in terms of costs and benefits
also think in terms of future costs/risks. even applies in things like preventing terrorist attacks. recoil if we put a dollar value on human life. but there is a limit of what we want to spend on that. can’t say “whatever it takes.”

exploit emergent phenomena

seek the root of intuition
steven’s talk: experts forgot what they learned.

study “wicked problems”
rittel and webber, “dilemmas n…”
then also, someone else wrote a series of blog posts on wicked problems
they have no stopping roles. solutions are “made the problem better or worse”, not true/false. no immediate test. one-shots. almost all wicked problems can be described as the symptom of another problem.
economics, epidemiology, spend some time looking at the techniques people have come up for wicked problems.

seek simplicity: don’t overcomplicate. but when faced with real complexity, grasp it: don’t oversimplify.

The Ruby Creation Myth – Sarah Mei minitalk at GoGaRuCo ’12

Sarah Mei’s minitalk at GoGaRuCo on Saturday, 9/15
still unedited!

If you read Ruby’s wikipedia page, you can see a bit about the history of Ruby.

1.5 years ago, went to the red dot ruby conf in singapore. Doing the tutorial . The organizer of the confernce had arranged to do a walking tour of Singapore with all the speaker. The only other speaker who was there for the walking tour was Matz! So Andy, Matz, and Sarah took a 3-hour walk around Singapore. =O She was at Ruby Kaigi in Tokyo, and had sorta met, but it was like “=O programming royalty”. 3 hours with Matz and can ask him anything I want!!!

C for over 20 years. Sarah said: the act of writing C and the act of writing Ruby feel very different. Resrouce issues, but also qualitative difference of how it feels. C developer –> Ruby exists. how does that happen? it feels like the birth of athena.
he thought about it for a few minutes and said, you know, i was writing ruby for a very long time before ruby existed. as a c developer starting in the 80s, he had starte dcustomizing his programming environment . typedefs, and macros, and each one was a very small change to solve a prrticular problem he had, but over time, these extensions grew. as a result, the c code he was wrigin started to look a lot like ruby. matz essentially implemented ruby as a dsl in c, which just blew my mind. did it over a fairly long period of time.

in the early 90s, turning it into a language was an incremental step in something that he’d been working on for a long time. lots of little good ideas, acted on each one, accumulated. changed the way sarah thought about programming languages and programmers and how you relate to people in the community. i’m not the person that has a grand idea and goes and works on it and code flows out of my fingers and TADA a wonderful thing appears. but having a bunch of little good ideas, one at a time, and acting on them as they

anyone in this room can be the next matz. or dhh if you want to be him.
(notetaker’s comment: this is so powerful.)

the genius that he has was really just in the process of noticing the small problems and fixing them as they happened over a long period of time. he did it over and over again. if you do that, you may realize one day that the small problems you’ve been solving are really component parts of one big problem. and your assembled solution suddenly seems like it can solve much bigger problems than you previously thought. then when that happens, you too can have an entourage that follows you around at conferences.

=)

celluloid/concurrency – Tony Arcieri minitalk – GoGaRuCo ’12

synchronous calls, just like you’re used to in ruby

asynchronous calls:
schedule work to be executed. not sure when. gets excecuted when
instead of bang, use .async, will get bang methods back in celluloid1.0
current thread returns immediately. same countdown will happen.
asynch calls just straight-through. never waiting for results. simple way to schedule work.

futures
best analogy: call ahead to a restaurant to order food, so hopefully it’s ready when you show up. maybe stil have to wait a few minutes, but better than waiting a long time.
fibonacci!
to get the value back, future.value blocks until it completes.
the messaging model is a bit

pools
group of actors. schedule work inside of it. call .pool on any class that includes celluloid. includes a class
calculate all those fibonacci numbers in parallel, get all results back.

beware the gil

contact:
@bascule
@celluloidrb
celluloid.io

blog: unlimitednovelty.com

Cruft & Technical Debt – Yehuda Katz – GoGaRuCo ’12

@wycats, wycats@gmail.com

yesterday, sandi metz’s talk was about how to avoid that cruft. this talk is what to do to fix it!

the gap bewteen the assumptions you made when you first made the code and years later consumes more and more development time.

sounds like classic technical debt, but it’s not — you accure technical debt on purpose, and imaging that you will pay it down.

the problem is like buying a lot of rim stock in 2008 and never
unlike technical debt, technical obsolescence will build up no matter how much you try to avoid it.

today, examples from open source ecosystem. open source solves problems generically. if there’s a gap, the gap widens quickly.
has given a couple of talks about things behind hard … bit.ly/harder-than-it-looks is a decent

examples:
jquery: dom readiness. one anchoring assumption stays the same, another is
rails: almost all of rails’s assumptions about security turned out to be wrong.
ember: data bindings, application structures. sometimes will have gaping chasm.

there are assumptions underlying every decision you make. the constraings under those assumptinos will make you happy today; when those assumptions are no longer true, that’s won eof the main reasons you as a developer becomes less happy about the code that you write.

this problem is worse with really great solution. with good solutions you can get away with having blah, bland assumptions, so the user is responsbiel for keeping that up to date, so . but i fyou have a really great solution, that takes the burden on itself, changes in assumptions have rapid effects on that gap. if, in rails, html rendering on the server is a bad assumption, then it’s not gonna be siply “no problem, tweak around the edges” — it would be a big problem. or if it turns out clientside dev is not a good idea,then game over for ember, because we took such a big bet.

side point: a lot of people don’t notice the difference between a partial and a full solution until the
so you end up with checkbox-oriented feature assessments — look at this library, does it have these features? — if you have a project that’s gonna be around for a long time, you have to plan for tech obsolescence. if you do checkbox, you’re not thinking about what assumptions underlie it.

a big objection to what he’s saying and hpilosphy in gneeral is:… you can avoid software
sandi’s argument is a weak version of this strawman: you can avoid all software problems by writing programs that do 1 thing and 1 thing well. if soemthing changes, remove that piece that viiolates the assumption and replace it with somehting that does,
i think writing good objecte oriented software is going to deal with some of the problems, but this misses the mark. this sort of thinking moves the complexity to a
instead of having a private api (what sandi called the omega mess), you end up with a public unstable mess
no matter how hard we try, these interfaces aren’t perfect. so the idea “don’t make omega messes” ends up with public apis that people using the system have to understand. moves complexity of edgec ases into components. pushes resolving complexity to user. integration tax being placed on user.

another interesting thing: people
write programs to handle text streams, because that is a universal interface. i’m not saying “ha ha i got you, node is not about text streams, game over” (laughter), there has to be a well-understood public interface in order for the idea of “write 1 thing that does 1 thing well” to work. cuz if you need to spend a lot of time

complexity
git
when you say text is a universal interface, there’s an assumption that people are using tab or some deliminated output.

early on in the stage of builing a car (19th century) people didn’t realy know waht a car was yet. so initially, the idea of a vertically integrated supply chain, ended up being effective. won the story early on because there’s the idea of a transaction path (pass?). when you’re not really sure what’s happening in between the pieces of the puzzle, if you do it all yoruself, you don’t have to worry how the parts guy is gonna talk to the assmebly guy.
over time, vertical integration does give way to distributed systems. this happens in economics. happens in response to more standardized way of thinking about htings and mroe supply chains.

how does this fit in with software?
we understand more about what it means to build a car or web app, know what it means
so a lot of the transaction costs become much cheaper. instead of large transaction costs, eventually it becomes clear (like how you integrate template functions in javascript.)
and integration gets cheaper … only some of the time. if you start off saying “frontload the whole thing! build standards on every link on the chain!” you’re gnona spend a whole bunch of time building standards that don’t end up mattering or really reducing transaction costs, will have become cheaper to not have bothered.

rack is good example of this lifecycle. giant vertically integrated supply chain. python did it before ruby did it; why, in the beginning of this process, did rails decide not to do that? rails had to choose between “what does it mean ” vs “spend a bunch of time to build a market place of 3rd party components”

usually small players that realize it’s annoying, as the little guy (sinatra, merb) to build all the connections. there’s a de facto standard. in the beginning, you want to build integrated solutions…

[…]

people respond to cruft in kneejerk fashion. criticism yields defensiveness. over time, in a project, especially as you have personnel turnover, the cruft that’s already there becomes received wisdom — “don’t touch that! it’s probably there for a reason!”

assumptions change.

example from jquery. dom node, want to put some data on it. if you put some data on the dom node then remove the node, gc never collects that data. bug in ie =(
proposed solution: put a number on that data. cache it somewhere. when we remove the node, we make sure we remove that data. we take care of it. easy to imagine “o it’s an old ie hack, probably 3 lines of code”…. but sometimes the assumptions you make about your code end up driving a lot of architectural decisions. as long as we support this old version of ie, or old microsoft project, we need this code. that’s where the gap comes from. means you’re gonna build up an immune system… zepto will come up and say “jquery you suck you should mremove that code!” and jquery is like “no if you remove it you’re gonna have a memory leak in ie”

a year ago, jquery did an analysis of the entire code base. they said, we didn’t do a good job of tracking assumptions, but we just analyzed what assumptions we made for old browsers and see . so we were able to determine the cost of removing ie. for a long time, the project had this general feeling / received wisdom that it wasn’t worth it to remove all the ie stuff. turns out it’s true after ie8.

lifecycle:
cruft = solution
cruft defensiveness
a gap appears
received wisdom consolidation. no matter how large the
splash of cold water (some new person comes into app, or releases competitior, turns out gap has become so wide that there’s some win for reexamining the assumption)
back to cruft=solution….

what’s a better lifecycle?
track your cruft!
in cruft.txt? maybe cruft.markdown? lol
write down a line of code, what the assumption that went into this line of code.
not possible to fully isolate all assumptions, you hvae to ship products, assumptions are real — but you can do this and say “is this assumption stll real?” ember: have recent chagnes in the last few years changed our assumptions? “we have to support old ie” is slowly fading.

on the flip side of that, it’s okay to be “the dom library for ie6 users”. it’s okay if you decide as a project that it’s a valid assumption for you.

this whole discussion is not just about the support matrix. lots of inputs go into decisions that you make

what do i want you to do?

1. i want you to keep track of the edge cases in your code. what the reasons that mght cause technical obsoleence.
2. and how they impact your architecture. think about how much of your architecture is based on these assumptinos.
3. periodically review your assumptions
4. make choices more intentional. document. help contributers make decisions without fear they’re walking on a grate.

Q: (josh susser) how about the test suite as a place to track cruft?
A: good place, and for every piece of cruft you should have a test, but my point is there should be a place/file where that’s the only job of that file.
Q: wrote a test to make sure the bug was still happening; when it didn’t happen, test would fail and it would tell you
A: good strategy for things like ie6 bugs. less for “microsoft project has this bug, some of our users are affected” … human assumptions, all the contractors still use fax machines. can’t erally write a test about it, but is an assumption.

Q: overfitting training data in machine learning. deny or connect?: cruft is one-to-one fit to universe at the time. in machine learning,
is that something we could use to help us write code? accept a less perfect solution for a longer-term solution?
A: you can accept that people will not always
it’s not that it’s perfect for some set of data, it is perfect for now. you do want to write code that’s perfect for now. if there’s a tradeoff of perfect for now or pretty good for later, you should take good for later, but a lot of times it’s whether it works at all now

Q: apple said “no flash on iphones.” when do you make that jump, for example, “no more ie6”? also how do you measure and come out with this data of how big the gap is, so smeone could make their decision for what to no longer support?
A: try not to have the answer never be “don’t touch that.” have the answer be “assumption valid or not”

Q: are you leading by example in this case, say, ember? or any project so we can see this in action?
A: no cruft.txt, but we do have architecture files that talk about why wer’e doing it that way. maybe if i have time. the lesson came from jquery where i saw how much work it was to engineer how to take the cruft out.

Schemas for the Real World – Carina Zona – GoGaRuCo ’12

developer and sex educator, think a lot about how these things overlap (xkcd about ‘sex’ not being found on fitocracy lol)

imagine everyone thinking that who you are and who you love is wrong. then imagine that everyone on the web doesn’t let you say who you are. excluded.

canonical set of relationship statuses?
3 years ago – facebook figured this was a pretty good model, and arguably
single, in a relationship, engaged, married, it’s complicated, open relationhip
separated, divorced, civil union, domestic partnership

g+ left out separated and divorced, and added ‘i don’t want to say’

allowing users to identify
driven by people rejecting a user experience that wasn’t driven by them.
not about monetizing — not providing

3 problems
1. the perception
2. the assumption that canonical lists for tehse things do and must exist
3. our faith that the first 2 problems can be solved by adding more list items. this isn’t gonna

so you end up with having to choose “in an open relationship with 1 person”
this is facebook failing to model relationships in a relational database. user has to choose between following along with the database or being truthful.

sam hughs – modern relationsal schema for marriage.
realized that relational databases just couldn’t model real life complexity. a graph database is needed.

how do we bring modern realities into the data, views, logic?
we can build a foundation for sane development and ux.

I. get schemas into alignment
1. mental schema
– set of preconcieved ideas
– framework for represeintg some aspect of the world
2. Database schemas
– mental schema translated into blueprint for database

when you use rails generate model or scaffold, it’s creating a migration. they ge ttranslated into unified schema to use in your database.

all this ux is just manifestations of mental schemas. our schemas and ux are leaving people behind, and we can fix it.

start with the question: what benefit will the user notice? this is NOT the same question as what will benefit the user, because the answer is ‘an awesome product, that will benefit them!”

1st: it’s shitty to be told “you are wrong about who you are”

MetaFilter – initially, developers said “dirty data is terrible” but then it was quickly embraced, hilarious posts (gender bender bo bender lol). says something about you, and what MetaFilter was used for. users asked, can we please share more? today, there are many freeform field, including ones where developers woudl instinctually validate. so values can be silly, nonsensical, even contradictory, — express yourself however you want to, we developers can handle it.

data doesn’t have to be for analysis.

diaspora* !!! sarah mei turned gender into a text field too! the users had fun.
some developers were not amused – effect on internationalization of gendered pronouns. gendered pronouns are a rat hole. so internationalization build on gender is bad. if you MUST, then ask the user what proouns you prefer. munroe (xkcd) says this. this matrix looks crazy, but better than others. (if you insist, there are some gems to check out — i refuse to RECOMMEND them, but here they are: SexMachine gem tries to predict male/female based on name and country LOL. dunno if there’s any amount of wrong in a gem that’s okay. other gem: I18n inflector gem, will fail)

as developers, we have this vision about what a good codebase should and should not be. we want structured, orderly, predictable. so we love stuff like relational, index, grabbable and sortable. make lists that are neat and exhaustive. we want clean easy analytics, so we can make decisions. but what we get when we try to do this with personal identity data is … tons of code. validations of what humans are like, then throw exceptions because they’re not. deal with conditionals and partials for meeting these conditions.

there is middle ground: a guided response comes in: autosuggest.
if necessary for structure, can try minimal suggest. for example, for gender, could suggest first 2 you think are most interesting, or first 7, or whatever, but if they start typing something different, then it becomes totally free form.

once they’re free to opt out of providing their personal information, of course they do. but the data quality will improve when users can choose waht they share. proven by metafilter’s experience. 40% of mefi’s users do use f, m, female, male.

facebook makes relationship status optinoal, but it’s coercive for people who do opt-in. most users do opt-in — 60% of them select some relationship status.

bottom line: we want users to feel passionate about their involvement of what we built.
analytics, in
the data being collected by coercive approaches: the danger is that it’s all bullshit, because it’s lying. conclusions drawn from coercive approaches may be fueling our decisions and it may all be wrong!

t.string “gender”
is how to solve that. make this stuff flexible up front. optimize for storage. decide what’s valid later. so that’s a discretionary field. optional,

t.string “relationship_status”, :null => true
documenting a product decision, a promise to your future self

takeaways

1. modeling the world is complex, that’s ok
2. assuming we know who users are surrenders opportunity to learn who they are!!!! this is innovation

Q: advertisers? what if they want
A: metafilter is financed by google ads. since there is no gender information (and mefi doesn’t run gender analysis), clearly google advertising doesn’t care that much, so it’s not an issue. you have the potential to learn who else is in this audience, you can start to address niches.

Q: not to diminish people who don’t fall into categories, but say for the case of a checkbox vs a text field, would you say that it’s a worse user experience for most people to have a text field rather than a checkbox or radio box? at that point, is it worth it for doing that?
A: all tradeoffs that have to be out of your user set’s needs, adn your app’s needs, you have to know what everyone tolerates. no way to make everyone happy, you will get people polarized over it. the earlier you do it, the less polarizing it is. mefi did it early, and h. diaspora wnet from a select to a text field so they

Q: you seem to be advocating this for all demographic data.
A: personal identi
Q: there seem to be some legal requirements. is personal identity data not part of the
A:

Q: i’ve worked with the dating space. a lot of users are gender normative and expecting that for matching. if you do open it up to a text field, how do you deal with gender-seeking? for 90% of the users expecting gender-normative answers.
A: auto-suggest. and the people who are not gender-normative are going to be excited to look for people who are more like them. wildcard search. spit out list of what other values are to use as tags for people to do some surfing around

Q: (sarah mei) hi. so i’m the programmer that made the change in diaspora. what i can tel you from our users: even people who do fall into gender-normative categories feel better about
a lot of people who identify with gender normative categories told me . “ada lovelace” as her gender. when you do this, you’re not reducing anybody’s user experience. you’re enriching everybody’s gender experience.

Services, scale, backgrounding – David Copeland – GoGaRuCo ’12

David Copeland, http://www.twitter.com/davetron5000
Based on his personal experience at LivingSocial, building Payments app. Story of rasonable programmers making reasonable decisions leading to WTF

Resque – creating a job that will be serviced by the NewPersonEvent class, given id when run, jam into redis (which is superfast key value store). So this will be fast, way faster than regular mail call.
Resque poller will find this class, will call perform, finds person in database and sends welcome email.
Now we can allocate things for usres to make , manage background process separately (even on a different machine). This is good, ability to scale under certain conditions, all good.

A few months later, we get an email about smoething that went wrong. This line of code inside our person controller (in ‘create’) generated a timeout. Resque.enqueue(NewPersonEvent

Tmeout to Redis
4 good things all came together to create a weird situation
1. signs up
2. doesn’t make it to redis, doesn’t get welcome meail
3. user tries to sign up again
4. can’t because of email uniqueness

Solution
1. person created and email sent
OR
2. no email sent and no person created. then we don’t have to go into console and fix it.
— So we will use database transactions to fix this.

Go back into controller. Person, activerecord, has transaction. If the resque fails, the user is never created; they still have to deal with a 500 error and sign up again. If we keep getting these timeouts, then we still have a problem, but at least it’s better.

A few weeks later, we get an exception in our job processing class.
In reque, when a job goes wrong, it goes into a fail queue so you can replay them. A transient error (like a teimout or network failure), you can replay that. This is not a transient error, it seems: person id not found in the database. But… wtf, we GOT the ID from the database? what could cause this?

1. got id from the database
2. event created. the second we create this event
3. event processed
4. BUT the transaction is not yet committed. the ability to roll it back is possible because no one outside of us can see what’s happening in the database. race condition. “and i see this all the time.” we can solve this by not using resque, who knows what other problems? other way to fix this?

could assume that if we get this sort of error (person not in database when this code fires) then it’s some sort of race condition. so we can assume that
ResqueRetryOTron7000 (assume we have something that does this for us)
have a condition that we expect might not be met at the time this is processed. if it’s not ready, try it again a fixed number of times so we don’t go into some infinite of
not amazingly clean beuatiful code but it does work and wasn’t that hard to complete.

better(ish)…
likely to complete in 5 retries
can replay any that bubble up (if this doesn’t solve something)
no grand rearchitecture required

our company: we’re so successful, we’re gonna buy our competitor.
we need to get their users into our database. our comptetittor didn’t have email validation.
we want same buisnes logic to happen as if they signed up on the website, but they’re not signing up–w e’re being given
another way to say it is:

after_create hook
every time a new person is created, do the new person
:after_create :new_person_event
controller code actually improves the codebase, is cleaner, which is rare (a new requirement actually cleanig up your code)

activerecord callbacks… maybe you don’t use them, but it’s hard to argue that it’s not terrible. it seems logical.

a few months later, we need to log stats.
1 method, 1 line, in send_new_person_event, add a log. Stats.ping(:new_person)

weeks later, need cache. add that in too. PersonCache.put(:name,@person.id,@person.name)

skunkworks project.. weird bayseian network… they have to add hteir lines of code in there as well.
…..

now you have a whole bunch of things ending up in our poor abused send_new_person_event.

Is this reasonable?
– it looks like a mess now
– but each line was _reasonable_ …
– that’s the definition of technical debt. we as a business were willing to accept this crappy looking method because we got business value more quickly. maybe someday we’ll fix it.

meantime… (parallelism in a company is just as crazy as in a computer) we were working on mailers. we decided we didn’t like them. migrating our old apps to use a new awesome mail service. they made it a simple REST call. “this is gonna be an easy migration”

so we go into NewPersonEvent. now we use MailerService.mail(:send_welcome_email,person)
if you’ve ever done this, you know it’s not this simple..

Meanwhile….. someone else is refactoring send_new_person_event. their solution is removing all those weird lines. they put it into the event itself. it’s running in an offline process where we dont’ care about if it takes longer to run. now we don’t have to mock those all out when we run our person tests. it seems logical that stuff about making a new person is in the new person event. sounds reasonable.

so we use the new mailer service, took care of technical debt, think everything’s good…. then an event fails.

1. mailerservice.mail(:send_welcome_email,person)
2. stats.ping
3. personcache.put
4. died. half-played… some of it has gone through, some of it has not; could fix in console but this is not good. we could wait for this line and hten do the next line, etc, but there’s a reason we’re not using node (chuckles…)

why does he think this is happening? mailservice is dumb. what would have made it smarter?
– idempotent calls (only sends 1 mail to the user, even if you call it again). how’s it gonna know which ones to send and which ones not to?
– additional calls to expose state

for making it idempotent (the way we’d implement the mail service not having to keep track)
– require a client-generated request id
– promise to only perform operation once per request id
so, can call mail a zilllion times, only one mail sent ever, same result goes back.

if the mail service instead couldn’t do that, but wanted to expose its internal state to us, could have done
– what emails have been sent?
-client checks first before sending.
not as good, but would have worked.

other changes were small, could have done within app. but this is changing the mail service, now we have a 3month old app going into version 2, sad.

what if
– mail fails at the gateway?
– i forget to check first?
– two request come in at the same time?
– my request ids clash?
jackie chan out of your mind! lol

So. The bad news is that you’re never gonna get it fixed. you can’t make it perfect. can’t prevent every weird thing frm happening. what you can do is get a better sense of what might happen so you can evaluate how to fixt hem.

ok, some strategies.

I. Historical record
1. have a historical record of what happened. log log log. rails is good at logging waht it’s doing, but you need to log your business information. what info would you loved to have a wrong. if it’s feeling expensive, do a debug and take debug out for production, but you want this.
2. audit activity. figure out who changed that, what did they change, when, maybe even why.
these two things together can help you piece together wha thappened at some point in your app’s history. often not order you think it’s gonna be.

II. prevent bad data
– AR validations canont be trusted. can’t rely on this only. you have to have this for usability. you know the database is gonna keep things in order.
– database constraints are very simplistic (mysql doesn’t have sophisiticated checks you can run)
– sanity check the rest (email someone if it finds something. run this in cron all the time. when it comes back with something, fix it instead of coding around it, will pay off down the road. make code have as few stupid complexities as possible)

III. fix errors
– email your team. “we gotta stop what we’re going and fix this problem right now. app has a bug, preventing user from doing something that’s iportant to business.”
– fix the problem or prevent the error
– downgrade to warnings as needed (so as not to fil your email inbox with unactionable items)

IV. extract services
– don’t just ‘be dum’
– design for idempotence – there’s a lot of complexity about preventing bad things from happening.
– provide help for client (so client knows what’s going on, can do a bit of selfhealing maybe)

i have really enjoyed solving these wacky problems, and livingsocial is hiring

i wrote this book that has nothing to do with this talk but i would love if you bought a copy of it (lol) http://pragprog.com/book/dccar/build-awesome-command-line-applications-in-ruby