I was reading an interesting study that Peter Norvig has mentioned on his website called "Comparing Java vs. C/C++. Efficiency Issues to Interpersonal Issues " which asked 38 programmers to implement versions of a program in C, C++, or Java. Another follow up study was conducted subsequently. I was curious to know how Lisp fared when compared with other languages. The results of the study showed that Lisp is comparable in its run-time efficiency to C and C++. The authors say, "this contradicts the conventional wisdom that Lisp is slow. Conventional wisdom is plain wrong." They also wonder why Lisp is not used as much as C and C++.
I started learning Lisp six months ago and I can now understand why Lisp is not used as much as C and C++. The programs that you write in Lisp look so different that people who have backgrounds in other programming languages wonder "what the heck is going on here?!" the first time they see a Lisp program. I have experienced that feeling myself when I started Lisping. The same is not true of programmers who are switching from C/C++ to Java or C++ to Python. Those languages have a definite "look and feel". The reason I think that is so is because they are all (mostly) imperative languages. Python is actually closer to Lisp than people realize but when you look at a Python program, you don't feel like you're looking at an ancient manuscript which modern day historians have failed to decipher. That is perhaps one of the reasons why people don't take the trouble to learn Lisp. I visited all the major bookstores in the city looking for Paul Graham's "ANSI Common Lisp". I never found it. Instead there were lots and lots of books about C, C++ ranging from "C++ for dummies" to "Professional C++".
Another reason may be because not a lot of teachers teach Lisp in Colleges. And even when it is taught, only a tiny portion of the language is taught and it is not given the rigorous treatment that languages like C and C++ receive. I have not seen any university insist that students implement a project of reasonable size and complexity in Lisp.
Different set of tools
Richard Feynman once said that the reason he succeeded in solving integrals while he was at MIT was because he had learnt how to differentiate under the integral sign while he was at high school. It is a method to evaluate integrals that was not normally taught to students. So whenever any integral became too hard to solve, they would give it to Feynman who would solve it become a superhero in their eyes.
In fact, he had a reputation for evaluating integrals when he was at MIT. And the reason, he admitted, was because he had a different box of tools. He knew all the usual ways of solving integrals- the ways everyone knew. But he also had this extra tool in his mental tool set. Lisp actually provides you with something similar. It actually provides you a whole set of new tools - things you cannot find in other languages.
The most useful tool I have come across till now is the treatment of functions like first-class-citizens. The way Lisp programmers think of data and functions is perhaps unique. They make no distinction at all. And no other language that is "mainstream" has yet succeeded in incorporating functions in data-structures like Lisp has. Also, there is a limit to the "closure" property that C and C++ try hard to achieve. In C/C++ you can have arrays of arrays. But after the first or second usage of closure, the code becomes messy and tangled. And a small error in a pointer assignment could crash your whole program or worse.
Java is no better. Though it is like a paradise for someone who has coded extensively in C++, because of its support for synchronization and threading and automatic garbage-collection, the language is too verbose. It also forces you into doing things the Object-Oriented way. So you are forced to crudely map situations (that don't naturally resolve themselves into situations that can be modeled as well defined objects interacting with one another ) into some UML mapping. And it has a very ugly way of dealing with multiple inheritances and fails to account for situations where object-orientation is superfluous.
I have observed something I have dubbed "language addiction" among programmers. Once they get really good at using a particular language, they are very reluctant to switch. And most who try to switch end up going back to their original language. This happens because sticking to what you know is easy and languages take time and effort - lots of effort to master. Anyone can start writing programs in C++ if they have programmed in C. But there is substantial difference between a seasoned C++ programmer and a novice C++ programmer who has not yet assimilated the object-oriented way of thinking.
The only way to gather new tools is by diving in and learning as many languages as you can. But you should be careful in choosing which languages to learn. (No Java programmer would learn C# willingly.) Learning languages that teach you to think in different 'paradigms' is better than learning languages simply because they are popular.