Open-source and its struggling Middle Class Tuesday, May 6 2014

Skip to the bottom for details about KeyHub.

I love creating libraries; building high-end components that other developers can leverage. Why? Because my efforts will enable more progress than I could directly create myself.

But open-source typically means free.

How do open-source library developers survive?

Here are a few common patterns I've seen in open-source projects.

  1. Weekender - A project small enough that occasional weekends by one developer can keep it maintained and high-quality.
  2. Orthogonal - A project whose development was funded by a OSS-friendly business with an unrelated profit model.
  3. Bazaar - A project with immense reach and a sufficient number of contributors that progress arises from the chaos. Linux used to be an example, although it's now also an example of #4 and #2.
  4. Support-based - A project sufficiently complicated to use/integrate that a large number of users will require support. Support contracts are the only viable option here; support incidents NEVER pay for themselves, let alone product development.
  5. Service bundling - Mozilla Foundation & Firefox are good examples of this; Google provides 85% of their annual income in exchange for being the default search engine. AWSSDK and Heroku Toolbelt are good examples of libraries with this model.
  6. Donation-ware - Wikipedia manages to sustain itself and MediaWiki through donations, although being the 6th most popular website in the world makes this quite a bit more feasible than your average app or library.

Weaknesses of these patterns for library development

  1. Weekender projects can't have a large scope; or if they do, they need to use low-friction technology. There are lots of Weekender micro-CMSes in Ruby, while very few are maintained for .NET.
  2. Orthogonal - By the nature of their orthogonality, there is little motivation for continued public maintenance after the first stable release. If properly advertised these sometimes transition to Bazaar, although more frequently they become Weekender projects or join the Unmaintained.
  3. Bazaar - Libraries require absolute backwards-compatibility and API consistency. While a large volume of pull requests is helpful, the lion's share of work still falls on the official maintainer even if they're only writing unit tests and documentation. Corporate sponsorship (Orthogonal) is nearly always required, but is rare in the .NET space.
  4. Support-based projects need a critical mass and a minimum quantity of enterprise eyeballs before the first dollar can arrive. There's a huge gap between Weekender and Support-based project sizes, making the transition impossible for most.
  5. Service bundling works great if you're offering SaaS interface libraries. If your library works 'offline' (as most do), this isn't really an option.
  6. Donation-ware needs lots of eyeballs to work. Libraries are naturally targeted to developers of a specific technology, which limits the user base too far. I've only seen a handful of libraries survive this way, and they were jQuery or Javascript libraries with extremely good advertising and millions of eyeballs.

New .NET OSS projects have fewer real patterns

  1. Weekender projects you fund yourself
  2. Orthogonal projects your boss or client has allowed you to open-source. You'll probably make a Weekender out if it, if it's small enough, and you care enough.

…but they don't scale

Neither of these options offers a profit model, neither has a path to scale. Getting them large enough to transition to Support-based can take 4-6 years (in my experience), and requires a massive financial investment.

For small projects a profit model isn't necessary. But when a project requires more than 25 hours from the maintainer per week, there's no light at the end of the tunnel. You can't offer support guarantees or contracts unless you're in control of your own time and employment.

Also - what if you WANT to focus on developing libraries full-time instead of orthogonal apps or services; what if library development is your goal?

Middle-class software is important; it's how platforms evolve

Solutions to tricky problems generally require a substantial amount of code. It's an inherent trait of software; the easy problems don't need libraries very badly, while the hard problems simply can't be solved without componentization. If you're solving a end-user need, there are easier paths to profitability. If you're just solving a 'tricky' or 'painful' developer need, you're probably in the software Middle Class.

Most progress happens in the middle class; our applications are limited by the quality of the platform stack they use. As libraries harden and evolve, they continually form new layers in the application stack, and have historically moved down into the OS, windowing system, and browser.

Unfortunately, Orthogonal software is rarely great quality. Focus is rarely on the API, but rather on solving the particular itch of the business itself. Most of the great advances I've seen have come from Weekender projects which took risky, radical approaches to solving big problems.

Open-source and .NET haven't been friends very long, and while there is definite growth from the Nordic regions and with certain applications (Umbraco, ServiceStack, Json.NET), CodePlex is overall a depressing graveyard of unmaintained, non-functional libraries that were once cared for and once worked.

There's a strong social component to the abandonment here as well; users of .NET OSS libraries tend to forget that the developer is doing this for free, and browsing the Issue Tracker or Discussions page will inevitably turn up absurd scenarios. I'm sure cultural differences are also a contributing factor here; what US developers consider rude, impolite, and demanding may be socially acceptable in some countries nearer to the 17th parallel.

How can middle-class software survive? Here's one ugly solution:

OSS purists might stone me, but I think the following compromise could enable hundreds of failing .NET OSS projects to survive and eventually transition to a Support-based model.

Enforced donation-ware; MIT-licensed source code with license keys

  • 'Official' source code and binaries will contain license key enforcement
  • Users will be legally permitted to remove the DRM and use it for free, and even branch the project and publish a DRM free version.
  • To preserve income, the official maintainer only needs to maintain better SEO ranking and better code quality than the derivative branches.
  • License key enforcement should be 'light' - just watermarking or displaying an advertisement.

How could this possibly work?

In any other language I doubt it would, however…

  • The .NET space is unique among communities for its general unwillingness to ever (re)build from source code or make edits to existing projects. We can turn this 'bad side' into an advantage.
  • .NET developers aren't typically morally opposed to license keys, they just wisely avoid using closed-source software with them. If presented correctly and clearly, I don't think many .NET developers would react badly to OSS+license keys.
  • .NET businesses are already conditioned to pay for components.
  • NuGet distributes binaries, not source code.

But how can you do DRM in open-source?

  • Like all DRM, it can be removed - just much easier
  • But with 256-bit asymmetric encryption, you can eliminate keygen cracks, making it impossible to trick the stock binaries or NuGet packages.

All you have to do is make the path of least resistance 'just buy it'. No legal mumbo-jumbo is needed, other than your Privacy Policy if you're using a central licensing server. Your license portal better support OpenID, offline keys, and floating domain licenses, however — the path of least resistance needs to actually be less resistance than deleting 5 lines of code.

I think this is a bad idea if you …

  • Aren't using .NET or something very similar in culture and community.
  • Already have a large user base. It's better to stretch for the support contract model than to risk alienating your entire customer base.
  • Have an alternative profit model

This is certainly better than…

  • Releasing closed-source
  • Writing your own license that's incompatible with OSS licenses.

KeyHub — lowering the barrier to entry

Licensing portals are notoriously painful — both for end-users and the maintainers. Typically, they're far out of reach for small companies, let alone individuals.

That's where KeyHub (Apache licensed) comes in.

KeyHub combines OpenId/OAuth login (no password required) with 2 great license key models: permanent offline keys (asymmetric encryption), and floating (online) keys. Keep everyone happy by letting them choose between convenience and offline support.

Our intended launch date was over a year ago, but due to severe issues with Azure, it got delayed yet again. It's now ready for alpha testing, and we'd like your help.

To make KeyHub successful, we need to find 7+ individuals or businesses that will commit to its use and maintainance.

If you're paying exorbiant fees for an ancient, closed-source licensing system, this is a great time to consider an exit. KeyHub is designed to support multiple vendors per application, and lends itself to a hosted scenario. Using shared infrastructure (CI, SQL, load balancing, fault tolerance, etc), we can make KeyHub both practical and reliable.

KeyHub offers a streamlined user and vendor experience that will reduce your pain. Join us in making keyhub.org a reality. Contact me, or send a pull request.

KeyHub screenshot

View comments on original page...

My experience with Azure Thursday, May 1 2014

TLDR; Azure deleted my company's data without warning and wouldn't fix it.

Let me start by saying that despite everything, I was extremely enthusiastic about Azure — until my last experience. Also, I'm mostly focused on git push style deployment (Azure Web Sites).

Forced off BizSpark plan by rounding errors

I was a late adopter of Azure (February 2013), after the new portal was released.

Roughly a week after switching to Azure, a rounding error killed all of my websites and databases. Despite having only empty databases, the system had decided I was using 1.000001 database units, .000001 in excess of my quota. There were no notifications; we found out about the outage the hard way - subscription shutdown. Despite being truly under the free quota, I just switched to pay-as-you-go billing and ate the cost. I couldn't get any help from Azure billing or support on the BizSpark subscription.

This is when I discovered that you cannot move services between subscriptions. If Microsoft messes up a subscription, you have to re-create everything — while your web presence is down.

Azure Web site deploy failure rate is high

My next hurdle was when I exceeded the 1GB quota for disk space with cache files. Azure web site storage being ephemeral and all, I'd expect it to just recycle and redeploy. Oops, no, everything just shuts down. There's also no way to resolve the issue (start, stop, redeploy doesn't help) I ended up having to move from Shared to Reserved to 'clean' the instance.

I've had about 20% of my deployments fail due to a DLL still being in-use. There doesn't seem to be any handling of file locking; the deploy didn't auto-roll-back either. Start, stop, redeploy, nothing could solve it. I had to downgrade to shared, then back to reserved so I can get the website back up on another instance.

Confidence - moving critical services to Azure

With that behind me, and impressed by the decent performance I was seeing, I gradually moved all my apps and sites to Azure, including web applications and customer databases.

Then this appeared in my inbox, timestamped 7:29am.

Your Subscripton Has Been Cancelled

I checked, and all of my azure sites were down. I logged into the portal, and discovered that all of my databases, backups, instances, and websites had been terminated and deleted. There was nothing left. In fact, everything had been deleted at midnight, 8 hours prior to the first (and only) notification of the action taken.

Now, the cancellation wasn't random, but it also wasn't completely my fault. Every year, Xbox Live bills me ~$120 MSFT for 2 gold subscriptions that I've never used (I always buy prepaid cards). It's a ghost that can't be killed; no record of it except the credit card charge.

A couple weeks prior to this event, a charge for $121.14 MSFT appeared on my card. I logged into my azure account and verified that my most recent statement was $38, and I had no pending invoices or e-mail receipts. After triple-checking it couldn't be Azure, I opened a dispute on the charge, certain it was from XBOX.

A few days after the cancellation, the matching Azure invoice appeared in my account.

Handling of the situation

I immediately contacted Azure support and explained the situation. They insisted that while invoices are generated some time before they're available for viewing, credit cards are not charged until afterwards. Perhaps this is how it's supposed to work, but it's not the order of events that happened to me - my azure charge was initiated before I had any warning. I wasn't expecting $98 of CPU usage either, which made the charge even more difficult to identify. I did re-authorize the charge with the credit card company as soon as I got the invoice.

The first representative did not offer to help resolve the situation or restore service to the subscription, but suggested I open yet another support ticket to migrate the services to a new subscription.

Due to the extensive delay, I redeployed to AppHarbor and restored service to my sites.

I continued inquiring into how to 'fix' the situation. I've included the transcript below.

Communication transcript

Hello Nathanael, Upon reviewing we see that all the data has been deleted as the subscription was disabled. Hence, data transfer is not possible. Kindly let me know if you have any questions.

Thanks and Regards, Aarish

Deletion of my data is violation of the Azure Usage agreement per 4c and 4e "Customer Data return and deletion". http://www.windowsazure.com/en-us/support/legal/subscription-agreement/ You may extract and/or delete Customer Data at any time. When a Subscription expires or terminates, we will retain any Customer Data you have not deleted for at least 90 days so that you may extract it, except for free trials, where we may delete Customer Data immediately without any retention period. You remain responsible for all storage and other applicable charges during this retention period. Following the expiration of this retention period, we will delete all Customer Data, including any cached or back-up copies, within 30 days of the end of the retention period. You agree that we have no additional obligation to continue to hold, export or return Customer Data and that we have no liability whatsoever for deletion of Customer Data pursuant to these terms. - Nathanael

Hello Nathanael,

Kindly be informed by design when a subscription is cancelled or disabled, all deployments are deleted leaving only the SQL database and Storage account which will be available for 90 days from the date of cancellation. Upon reviewing the usage, we see that SQL database usage stopped on 9th July and the subscription was disabled on 10th July. Let me know if you have any questions.

Thanks and Regards, Aarish

However, you're saying that the SQL and Storage accounts were deleted.

-Nathaanel

Hello Nathanael,

Thank you for your response. The SQL database was deleted from the management portal before the subscription was disabled from our backend. Also, there were no storage account usage for this subscription. Hence, no data was retained. Let me know if you have any questions.

Thanks and Regards, Aarish

At which point I took a day off to cool down. I certainly didn't log in and delete my own data on the same day my subscription was terminated. I can only assume that the audit logs for subscription cancellation were simply confusing.

Hello Nathanael,

I haven’t heard from you, and hope this means I have completely addressed your concern. If you do have any further questions, please reply and let me know – I will be happy to assist you further. If not kindly let me know if I have your permission to archive this case.

Thanks and Regards, Aarish

Hello Nathanael,

I still haven’t heard back from you, but would like to confirm that the issue is fully addressed before I archive this case. Please let me know if you have any additional questions or concerns.

Thanks and Regards, Aarish

Apparently waiting a couple of days wasn't enough for me to resume the conversation with a polite tone, which I feel bad about.

Surely you're joking? What have you possibly resolved? - Nathanael

Hello Nathanael,

Kindly be informed that as there were no SQL database and storage account at the time when your subscription was disabled. No data was retained. Hence, we are not able to transfer as per your request. Let me know if you have any questions.

I apologize for the inconvenience.

Thanks and Regards, Aarish

I'm pretty sure that DB still had data at the time the subscription was cancelled. Also, why can't you at least transfer the Website configurations to the new subscription?

-Nathanael

Several days into the conversaion, Aarish finally offers to re-enable the subscription (but explains that it's a useless effort).

Hello Nathanael, Kindly be informed that there are certain prerequisites to perform data transfer. Some are listed below;

1: Source and destination subscription has to be active. 2: Destination subscription should be empty. 3: Service administrator of both the source and destination subscription should be common ( at least until the transfer is complete).

I have enabled the source subscription from backend. Kindly login to the portal to check if the website is available. Kindly let me know if we can schedule a call to discuss further.

Thanks and Regards, Aarish

So, you're saying that since I already have 1 server using the new subscription, I can't move anything to it? This kind of limitation needs to be communicated much more clearly to customers. - Nathanael

Hello Nathanael,

You are correct. If the destination subscription has any services running, we cannot move services from another subscription. Also be informed that the source subscription was enabled from the backend, kindly verify the same and let me know if you have any questions.

Thanks and Regards, Aarish

Results

Many users of ImageResizer have asked for a SaaS version of the software. I wish I could provide it to them, but I cannot entrust my users to a host that capriciousouly deletes customer data. It's also not cost-effective run Windows on AWS, which means an ImageResizer SaaS is not happening.

Azure has potential, but Microsoft has never been good at billing customers accurately or resolving related issues. I've been using AWS and Heroku for years without experiencing a single billing-related service disruption — despite having credit cards expire multiple times.

Don't assume I have perfect faith in AWS, Heroku, or even AppHarbor. The only decent way to mitiage risk is plan for multi-cloud support. If you can redeploy to another provider, you can avoid bad situations. Unfortunately, AppHarbor is the closest thing Azure has to competition.

What Azure needs to improve on

  • Use a different merchant description for Azure charges.
  • Consider rounding fractional amounts down instead of up. Floating point math is hard, we know.
  • Notifiy customers before (not after!) deleting their stuff.
  • Perhaps, if there is a billing issue, give customers a time window to resolve it.
  • Add a "Reset" button to websites that recycles the VM and ephemeral storage, so when code fails, users can rescue bad situations.
  • Be transactional about website deployments. If you couldn't deploy all the files, roll back.
  • Handle locked files.
  • Provide invoices via e-mail.
  • Don't be arbitrary and annoying about subscriptions. Let them be repaired.
  • Try less hard to cancel customer accounts.
  • Empower customers to resolve subscription problems without involving support.

We love you AppHarbor, stick around!

And last, but most importantly — Azure needs viable competitors. With linux, we can pick and choose cloud providers at will, but investing in a Windows-based cloud application ties us firmly to Microsoft (and their strengths and weaknesses). If Windows is to be a viable platform in the future, it's cloud story must be multi-vendor — in a way that it isn't now.

Microsoft has invested in their competitors before - and this is a great time to do it again. Azure can't survive alone.

View comments on original page...

The War On Waste - or Why I Focus On Libraries Saturday, November 2 2013

Skip this if you're not in a philosophical mood. It's a 1000-word soliloquy about personal goals and regrets.

Also, despite the next section, I am pro-Microsoft. My personal goal is to make developers happier and more effective on whichever platform(s) allow me to do so. Also, vendor loyalism in software often hurts the vendor - insular feedback is tremendously destructive.

Background — Waste

During my pre-2008 loyalist phase, I wasted precious years struggling with sub-par tools and writing mountains of boilerplate code. I trusted my peers (and even — foolishly — technical evangelists who've since moved on) to know which tools were best — after all, they'd been in the business much longer than I.

In defense of my peers and I; when one works exclusively on a platform of black boxes, guessing only through inference of results how they operate, one tends to stop questioning the manufacturer's guidance. Belief systems usually form, evidenced by code rituals and cargo-cult behavior. Shotgun programming proliferates, and manufacturer advertising is absorbed as if it were truth. Logic, without good data, is useless to every mind. Software, without source code, is an endless quantity of bad data. Demand source code for everything you touch. As the depth of the software stack increases, vendors who don't provide source code and transparency will reach a paralysis point.

Our planet is being revolutionized by software, but in a horrifically primitive way. Man-decades (sometimes centuries) of human creativity — per business — are spent to reinvent line-of-business software, which is used improperly (if at all) by a handful of people, and thrown away after a few years. Meanwhile, projects and organizations with a demonstrably positive influence on the world struggle to accomplish their (often unique) software goals. Despite an explosion in the number of software developers in the last two decades, very little has changed about what wastes our time and energy. For a profession that builds its own tools, it's extremely shaming that leaky abstractions are still so prevalent and that code reuse is so rare.

Software has a human cost

As developers, we tend to pour our mental energy and health into our work, leaving us so mentally drained that we are often ineffective at life itself.

I admire and envy those who balance work and life easily, who can leave problem-solving at work. I've never been able to. I shower thinking of algorithms. I shave while trying to simplify interfaces or APIs. To expel a problem from my brain, I have to read an extremely absorbing book — or replace it with a better problem.

Based on those I've met, I don't think a healthy balance is the norm for software engineers.

Software has a human cost, but that cost is different for everyone.

For me, it first struck after 4 years as a paid developer. I was 18. I take 4 medications and 5 supplements every morning as a result. My eyesight is slowly worsening. My daily visits to the gym are driven by my desperation to recover; to be around when my son is grown.

Minutes in which I have mental energy to expend are precious to me now; I don't take an hour of coding for granted.

No sum of money could justify what I (or countless others) have put into software. Our best hope to balance the scales is to translate our past, present, and future time into as much net happiness for others as we can.

Open source

I can't force the world to adopt or create open-source. Economics (at best) suggests a slow erosion of proprietary systems, but at the current rate it could be decades before we see code reuse exceed duplication — if such a thing occurs.

The sheer demand for software developers is also two-edged sword. It raises salaries, but also makes it that much harder to fund software that doesn't generate revenue.

Open-source tackles waste head-on, often fighting against massive advertising budgets with only agility and developer altruism. And somehow, it's winning against more wasteful models.

I think many confuse the enemy here. It's not Microsoft, not Apple, not closed-source software. It's waste itself. Our enemy is the squandering of human creativity, the art burning in piles of discarded one-use software.

To me, the Apache and MIT/BSD licenses are the most effective at reducing waste.

I can't find the GPL attractive. What benefit can be had by preventing closed-source programs from using your library? Developers aren't in charge of business decisions; they're just trying to get their job done. They're probably more grateful than most for what you've built, and almost as likely to contribute back. Excluding the average developer doesn't help you, them, or anyone.

There is no war between open and closed source software; only against waste.

What drives me

Lacking influence or riches, I focus on building tools.

I patch (and build) libraries to save developers time and frustration. You should, too.

Good ideas are contagious, even when implemented in the briefest manner possible. Show just one person what could be, and soon it will be what is.

Simplify the complex. Eliminate leaky abstractions. Reduce code noise. Automate the mundane.

Help developers focus on the unique parts of their challenges, and you've multiplied how much they can accomplish with their time.

Remove useless disruptions so they can spend more time in the flow, and you've made them happier people (clinically proven). Flow is one of the primary causes of happiness; leaky abstractions make it harder to be happy.

This is a goal every developer can achieve. Fixing a bug might save 100 people 30 minutes of frustration. That's a massive win in terms of net happiness, even if it was hard for you.

Just the psychological effect of a pull request is enormous to a library developer. It's a concrete validation that their library is useful, worthwhile. And as an extension, that they have been.

Select your tools with ruthlessness; repair them with consideration. Require source code to everything you touch. Optimize for net happiness.

View comments on original page...

Browse archives...

About Nathanael

Nathanael Jones is a software engineer, father, consultant, and computer linguist with unreasonably high expectations of inanimate objects. He refines .NET, ruby, and javascript libraries full-time at Imazen, but can often be found on stack overflow or participating in W3C community groups.

ImageResizer

If you develop websites, and those websites have images, ImageResizer can make your life much eaiser. Find out more at imageresizing.net.

Imazen

I run Imazen, a tiny software company that specializes in web-based image processing and other difficult engineering problems. I spend most of my time writing image-processing code in C#, web apps in Ruby, and documentation in Markdown. Check out some of my current projects.

More articles