HDE Advent Calendar 2015Merry Christmas! This is Xudong.
I was looking through the topic list of HDE Advent Calendar 2015 and found that there is really not much introduction about how working in HDE is like. Isn't it weird to have a blog for PR without much content about what people are working on or how they are doing it? It should not be a bad idea to share some of my experience as a developer here this year.
I joined HDE this March as a developer and was assigned as the 4th member to a development team working on mail archiving feature of our HDE One service. The mail archiving feature is one of those features that existed since the begining of our service and it has a code base with a history of over 5 years since our CTO bootstrapped it. The development team working on this feature has always consisted around 3 ~ 4 developers and most of the development is about adding new components to enhance the feature. Within these years, the code base for this feature had grown from some standalone daemon in Erlang to a repository that contains core libraries, several daemons, some HTTP API server and some web applications. It contains huge amount of code in Python, Erlang, Ruby, HTML, Javascript, and also shell script to put them together.
So here my journey to deal with this code base started.
My main task this year was to rewrite some components written in Erlang in another language -- Golang. It was a huge task to me as I had never used these two languages for any serious project before and I was not familiar with this code base. Luckily the overall logic of the components is not too complex to handle. And there is still someone who remembers some details of the implementation. (Thanks to all colleagues in my team!)
I managed to separate the main task into these subtasks and managed to finish them as planned.
Understand the data model of original components.
All the components share the same data storage and we use a non-relational data model on AWS storage. It is quite self-explanatory and straight forward. After some time spent on reading the code of data access layer in Erlang, I was able to reimplement it easily with the help of AWS SDK for Golang Although the existing cache policy is a little quirky, to make less trouble, I decided to implement the policy as close as the original one.
Understand interfaces that interact with other components in the system.
The components use standard protocols like HTTP and SMTP with customized extension to interact with other components. So it was quite easy to understand and implement with the knowledge of these protocols. I managed to enjoy the dark magic of implementing RFCs.
Reimplement data conversion logic.
The core logic of components contains some kind of data processing pipeline to convert data in one format into another format. So some kind of text and binary data parser and generator are required. Luckily, the document of the data format definition is in the code base. And I didn't have to check much of existing Erlang conversion logic as I knew I would not enjoy it. With this definition in hand, there is little difficulty in writing new parser and generator in Golang.
Reimplement core logic.
This is the most problematic part as logic in Erlang is expressed in some functional way while it should be converted into Golang's structural way. Also, some of the code in Erlang depends on Erlang's internal data representation and conversion rules. I had no choice but to reimplement these rules in Golang. It involved a huge bunch of bits and bytes manipulation. Whatsoever, after going through some thick and thin, I managed to finished all the reimplementation successfully.
Migrate existing tests and add more tests.
When doing some rewriting, this is really the most important part. Although our test coverage of Erlang was not ideal and the layering and structure of code are not identical between Erlang version and Golang version, it is possible to migrate most of unit tests to the new one. Especially tests over the data conversion part.
I did the migration of tests at the same time when I was rewriting the code that is under those tests and added more tests when I found necessary. After all these tests were implemented, I was highly confident about the correctness of my code when all test results went green.
The reimplementation tooks around 6 months in total, during which time I was also involved into other works. Although usually developers hate rewriting, I found that this reimplementation work is quite rewarding.
During this time, I have learned more about:
- Golang and Erlang
- AWS services
- the whole work flow from analyzing legacy code to releasing some artifact
- Some commonly used protocols
And I also get to know more about the code base my team is working on. What is more, it just feels good to see the new shining components growing up from the old code base.
