Competing with Craigslist

A number of startups have contacted me with plays to compete with Craigslist. Craigslist is dominant in providing localized exchange of goods and services. They are free, they have a huge brand and audience, and (from the outside) they appear to be a relatively efficient organization. The competing startups are either trying to out Craigslist Craigslist or they are going after parts of the whole by focusing on a particular commerce area, like rentals. Those that are going whole hog had better have not only superior features for entire range of commerce that Craigslist covers, but have to be just as consistent as Craigslist is in their usability. And Craigslist’s minimalist design, while it doesn’t do much, it’s easy to get and it’s consistent across the site. It’s a simple form for posting all types of requests — almost always the same no matter what is being exchanged. For a company to take it to a higher level of use and functionality a significant investment is required, and so far I don’t see any threats to Craigslist of this sort.

A better strategy I’m seeing now is from the startups that are those focused on just a particular commerce segment and doing it better than Craigslist. In the long run, they have a better chance in that they are better and free to the consumer. The “free” part of course is a tough business decision at any time, and especially now, but against Craigslist you have little choice. A couple of interesting players in this arena are: 11squarefeet which focuses on a new sub market of renting: office and desk subleasing, and okCupid which is free online dating done extremely well. Okcupid is alot of work and thus must have relatively high development costs so it remains to be seen if they can exist, but they can give Craigslist free dating a run, let alone Match, eHarmony, et. al. (Since an automated dating site ( matched me up with my ex-wife (really), I’ve been more than cynical of these…).

It remains to be seen whether an entrenched player like Craigslist can be removed by the strategy de jour: a mashup + better usability + a market-focus = success. Or do we really need to start investing in harder technology that raises the bar of service and efficiency. An example of such investment I’ve seen is Siri which attempts to up what can be done by using bleeding edge technology. This is an example of a number of AI-ish plays that are seeking to leverage advances in NLP . We may be reaching the end of Web2.0 when the strategy du jour is no longer so easy to succeed with. With that said, Craigslist, by being community focused, comprehensive, and free, is a tough combination to best by any means.


Testing the Frontend: Automating Web & Javascript Testing

Testing javascript heavy, web frontends remains a hard problem in software development. Everyone wants to do it in order to be agile, but few if none can really do an ideal job of it. Some key reasons they want to automate are:

  • allow developers to quickly smoke test their work before they commit it
  • support extensive, comprehensive functional testing

The reasons it’s hard to do are twofold 1. the frontend in a new product development is always in flux, and 2. there’s no great tool for quickly creating robust test scripts that can be automated. In particular, while there are some recorders for the gestures on the frontend, they usually fail in certain ways that demand one to further generate custom test scripts for that tool to run. It’s extra work that often becomes outdated on the very next build.

Generally, good agile practice demands lots of test code to support quick and reliable bug finding and fixing early in the development cycle, and not sometime later. So good teams will make some compromises to their agile test strategy to get as close as is reasonable to ideal without it becoming non-agile. The typical strategies are:

  • create unit tests for all infrastructure, including going up into the javascript client as far as possible, but not automating user interaction
  • create testable javacript widgets (look and feel) that can be instantiated into test pages that can be quickly looked at by a developer or tester

Some tools that help automated javascript and RIA testing are:

  • Windmill
  • Selenium
    • Provides multi-platform testing and recording
    • Downside: recorder doesn’t handle all circumstances so programming is required, and test scripts will end up with dependencies on DOM structure
  • Dojo
    • Dojo is a javascript library with testable widgets, and a testing infrastructure for scripts
    • Provides an experimental testing infrastructure, DOH, with a recorder

    All three provide recorders. though Selenium is probably the most mature. They all suffer from issues in recording where the auto-generated scripts can often fail, require knowledge of XPATH to sort things out, and are not easy to incorporate into development processes so these scripts are maintained easy.

    The key thing here is that it’s tough to incorporate these tools, which are themselves less than ideal, into workable processes that developers want to do and that support the flexibilty product designers need on the frontend. The upside is that if a working compromise can be found, the development will ultimately go faster & cheaper, because issues can be found sooner.

    The long and the short of it is that if you want to do an agile process, it takes time and planning but the results can pay off. In the end, simplicity on the frontend pays off not only with users (typically) but also in development costs and time.

    Limitations of Trac

    For one of the startups I’ve been involved with we used Trac as a quick way to support “development”. But Trac is very limited and problematic. We choose because it’s free, and a number of successful opensource projects use it. It works pretty well for supporting developers and an open environment, but as we’re running a startup with a rag-tag fleet of programmers, testers, project and product management it’s really not sufficient for all of them. Out of the box, Trac is best for developers and developer run projects. But if you’re a product company, then you’ve got alot of other things to manage other than what the bugs are and which developer is working on them. Because that’s really all you get. When you have people in other roles than programmer and technical manager, you’re beyond the capability of Trac: there simply not enough fields and capabilities to support different kinds of development roles and their interactions.

    Perspectives in Technical Due Diligence Consulting

    The key to performing a technical due diligence is an understanding of the business goals that are driving the audit. To perform the audit well requires discipline to listen and the knowledge to ask the right questions that will uncover what is going on. A simple methodology is helpful as a guide to obtain information without bias, to remember to ask all the questions that should be asked to get a fuller picture — you don’t want to dwell on one sore spot or mine only one’s particular interest. Knowledgeable listening and experienced objectivity is key to not getting lost in the expectations and projections of the sponsor rather than the reality of situation. In the end, the client is happiest when the truth is told, even if it’s not what they want to hear. Sometimes it’s worse than the sponsor expects and sometime not.

    Kinds of Technical Due Diligence

    Due Diligence for an investment interest is different than that for a CEO or division head who is worried about the performance of an organization. All organizations have problems, but some matter more than others with respect to the goals of audit.

    Sometimes the outcome of a due diligence has been that I am put in charge of fixing it, but that is not my intent in taking on a due diligence. I am after providing the best insight into what is going on, what is working and what isn’t, and what isn’t so clear, and what of all that really matters to the business and what’s to be done about it.

    I’ve been on both the buyer’s side and the sellers side of M&A conversations and in many other angles that drive a due diligence assessment. I know what the various players are looking for and what they are wanting and avoiding. This (advice from someone being assessed I find problematic in that it’s more about defending oneself than learning. A decent due diligence should be constructive for all concerned. It certainly depends on the situation, sometimes the agenda is loaded and sometimes it is merely exploratory… and in the end transparent truth is best, for it serves the greater good.

    Acquisition Perspective

    In an acqusition driven audit, the team being evaluated often has much at stake, and defensiveness can come into play. However, it is best to work closely with the auditor, because that constructive play can be the best card to play. A team that represents itself as knowledgeable and effective in an audit is not only being the better corporate citizen but also serving it’s own best interests.

    Investor Perspective

    Buyers are after ensuring their investment is all they expect. They are looking to avoid buying problems or at least to understand them enough to price them. This requires a bit of digging since the Buyer isn’t incented to display their issues. The role of the assessor is to dig in this case, but in this case there is a win-win in the assessment. The buyer already sees value in the company, so the assessor has the opportunity to help identify with the team being assessed the challenges ahead and how to address them. The question is can they or what will it take, and with what impact. If they are worth buying they will be able to face these challenges. The assessor should be able to help the seller and enable and identify a successful relationship between investor and invested company

    Seller’s Perspective

    Are looking to maximize value. They need to be showing their real issues rather than hiding bad surprises for later, and they need to show how they are constructively addressing these issues.

    Team in Trouble

    This is always tough because these teams can be embattled. Change is imminent. Due Diligence means that something isn’t working and things have to change. But the upside is that aim is for mutual success. People in this situation typically are doing their best to fix things or to survive a tough situation. It’s not a happy occasion, but there’s lots of pride at work, and that can be a powerful factor. Engineers like to ship real working code, and when it’s not happening, there can be many factors at play. The assessor must stay focused on the business goals, and is looking also to enable the values of the business. At times this might seem cold to a team, as in the icy ‘it’s just business‘, and at other times it will seem senstive to the considerable work and care teams typically put into their work. Nonetheless, a profitable and successful enterprise has to focus on the bottom line (be it a single, double, or triple bottom line), so the truth must be spoken. Like in hiring, if something doesn’t fit, it’s best to change it rather than hope to change the person… it’s better for all concerned, eventually. But often, changes to process is what is needed, validation of what the team already knows and helping them get empowered to act is often the cure. I will often not only make recommendation to the team, but to the senior management as well who are often part of the problem.

    Commitment in Early Stage Startups

    We’re working with a rag-tag fleet of development folks: programmers, admins, designers and so forth. Amongst the challenges is that as we bootstrap our development it is a huge challenge for the folks we’re working with to be as committed as we are. For many of them this work is part-time and they squeeze into their busy lives. On top of that, they are distributed all over the place so that keeping everyone in sync and focused is a challenge. And the remoteness itself, without having worked together, makes it very hard for us to develop a solid team feeling.

    We stress respectful communications. And we really can’t over communicate in this situation, have too many meetings. Not yet anyway.

    Some of the relationships work really well at a very high level of professional commitment, and others are not so engaged and we move on. We’re looking for folks who passionately involve themselves.

    Big and Small Scale Development

    I’m reflecting on the scale and speed of changes in my work of late. During this period, I’ve been managing a team of 200 people and a very large IT budget, with all the issues of a large organization, enterprise scale technology, and the pressure of a market leading position. There’s the politics, the multiple agendas, and battle against crustification inherent in such an organization. And there’s the pleasure of a large, well-funded team accomplishing many many things at once. And that’s been fantastically fun.

    More recently, I’ve been working with really small teams, including my own startup, where I’m am doing what has to be done just as before. But now it’s the writing of code, working with loosely connected remotely distributed developers with the blistering urgency and minimal resources of a startup. And I’m enjoying that immensely too. And each of these experiences are informing the other. Each are different sorts of challenges related to scale, none better than the other.

    HTTP Server Push: State of the Art, and the State of Cometd

    The Problem: Web apps often require immediate communications between users mediated by the website, rather than directly connecting users to each other. That website communication via HTTP requires the browser to poll the servers, servers cannot push data to browsers. An example is a chat application: one user wants to send a message to another, so their browser app sends it to the server, but the recipient doesn’t know that there’s a message there on the server to be picked up. The server can’t push it to the recipient’s browser. So the problem is how can the recipient recieve messages quickly and reliably?

    The solution is a set of hacks that are collectively known as ‘Comet”. Comet actually a combination of techniques intended to work portably across browses as efficiently as possible, and to work efficiently on the server-side as well. It’s basically, a bunch of ways to do efficient polling. And the reality of it is that at this time there are a bunch of technical challenges to implementing them.

    On the Server Side:
    The assumption of most http (web) servers is that a browser connection is quick. Most web implementations use interpreted languages which take up alot of memory (ram). So with quick connections of bloated memory, you don’t use up much memory. But if a connection is to be long-lived, then web server will have many more bloated interpreted apps to run at once and will run out of memory sooner. So you get a scaling problem.

    You either need smaller apps that take less memory (so you’d have to forget about the entire “P” in the LAMP stack, or you need a server that can run a more efficient server to handle push efficiently.

    There is a class of servers out there know as CometD for such long-lived connections. The state of Cometd servers is experimental at best. They are basically bodies of server code which you have to hack. There are no standards, most don’t have plug-ins or any clean way to isolate your code except through coding discipline… There’s no mod_cometd for instance for apache etc. So you’ve got to hack your own. It’s not rocket science, but it’s certainly neither enterprise quality or highest-level maturity, open source quality. It’s the first round of sketches… You have fill in here and there, but you can get the general ideal working.

    CometD really only specs a transmission protocol known as Bayeux. There’s no server or client standard, only the protocol. The Bayeux protocol describes the transmission of data in a publish/subscribe model. There is no further control or information for the server to something other than broadcast messages from a publisher to a multiplicity of subscribers. Great!, but say you don’t want just anyone to subscribe to your channel… well the protocol isn’t going to help you directly. So a basic CometD server that supports Bayeux does nothing other than route messages, anything else you’re going to have to provide through modifications to the server. And as I noted there’s no standard here, not nice plugin to modify the server… You have to start hacking. So be it.

    – no security: there’s currently no means to prevent anyone from subscribing or publishing… You have to hack that in yourself.
    – no connection to your App or DB: you have roll your own. There’s no pre-built mechanism like a callback that you can hook into on certain server events. You have to read the server code and roll your own system These shortcoming are being addressed more or less in open-source and in commercial server implementations. Less for now.

    Check these out as they’re the best way to get push done:

    * From Facebook Engineering blog: For Facebook Chat, we rolled our own … epoll-driven web server (in Erlang) that holds online users’ conversations in-memory and serves the long-polled HTTP requests. … Why Erlang? In short, because the problem domain fits Erlang like a glove. Erlang is a functional concurrency-oriented language with extremely low-weight user-space “processes”, share-nothing message-passing semantics, built-in distribution, and a “crash and recover” philosophy proven by two decades of deployment on large soft-realtime production systems.

    On the Client side, there are Comet libraries that implement the Bayeux protocol to CometD servers, but given the state of the protocol, you’re only getting a starting point for significant elaboration. Dojo supplies one of them, authored by Alex Russell, who coined the term “Comet” and who is one very smart and able technologist.