Next-generation software development tools
May 4, 2011, ESC, San Jose, CA– Dan O’Dowd, CEO of Green Hills Software describe the need for software developers to upgrade their tools. A revolution in software development tools is overdue, because programmers spend most of their time tracking and fixing difficult problems.
Development tools are not evolving to match the state-of-the-art. In a recent survey, 61 percent of programmers are still debugging programs using “print” statements. These are the same type of tools O’Dowd used back in ’77 to design software for a handheld football game for Mattel. At that time no embedded software developer tools existed, so he invented a GDD type debug tool.
Lesson here is to always insist on the best software development tools possible. If they don’t exist, make them. Programmers are tasked to implement all the features that optimize performance of the whole system and are often negating item on a company’s ability to ship. In addition to the required features and functions, the programmers are also responsible for security, reliability, and cost of the final system.
One factor to consider is the investments companies make for some of their key workers. Automakers spend about $100,000 per worker to manufacture their products. Trucking companies spend a similar amount for the trucks and support systems for their drivers. And electronics engineers have access to more than hundred thousand dollars per person for software, tools, and hardware for doing designs.
Unfortunately, embedded systems programmers get only a small fraction of these dollar amounts per engineer. One notable exception is Green Hills, which spends about one and a half times a developer’s salary for tools and support. Overall however software developers are not getting financial support for their efforts. In most companies, the greatest costs are for people and total costs run about $200,000 per developer per year.
Programmers need to get the latest computers with sufficient compute power and memory to do builds in best of class times, around 15 seconds. The developers need lots of big monitors so they don’t have to try to find some small window buried in a single screen. The infrastructure should support fast download times and they should have access to hardware for prototypes and lots of additional memory and storage for debug.
Some of the tools developers should have include hardware debug probes, trace probes, which are like seatbelts in that he only need them when something goes wrong. They need to have tools for performance and static analysis to help find the more common mistakes. Some of the worst bugs are related to memory corruption, and the tools exist to find 90% of memory bugs. A related problem is memory leaks, which are due to full-time use an allocation creep, facets of operation which are not easily tested in development.
Intermittent failures by definition are hard to find and fix because they don’t happen all the time. A standard practice is to set breakpoints to trigger when timer change errors and hope to catch timing errors. A better solution is to use the trace probe which captures all memory, instruction execution, and memory read and write operations and puts them into storage when an intermittent failure occurs you can stop the recording and run the code backwards until you find the cause of the error. Trace probes eliminate the problem of using other debug tools, which can move the failure point.
Existing correction tools and practices are like CSI, you look for clues, collect data, create hypotheses, and confirm with correlation to the data. With more advanced tools it’s like having multiple cameras and recorders looking everywhere at the time an event occurs. Embedded programmers can improve their productivity by getting access to, and using better tools. Many developers are still using 20-year-old development tools and technologies.
An area of growing importance in embedded systems is security. The lowest level software needs to be secure, so developers must evaluate their baseline software and build a secure layers on top of that secure base. Unfortunately, this becomes a productivity issue since over 57 percent of all programs are delivered late.
