Why I didn’t dropped out

Ok so, first of all why at all a thought of dropping out of college? Why Do I or why I did thought about ditching the degree of which millions aspire every year and is awarded to select few. This degree which holds lots of value and anywhere I go with this degree, I would have a more than decent level of respect and admiration for. I’m doing my undergraduation in Civil Engineering at Indian Institute of Technology Bombay – Ranked no 1 in India and ~30 in World.

Well this all is one of the reason, leaving all these pride factors and convincing the stereotypical society is not easy. This is not a huge reason, if I’m asked to give weight to this reason I would say 1/100. I really don’t care about the stereotypes that exists in this society.

I don’t like Civil Engineering and after studying it for 3 years I have “NO Idea” what it is like to be a civil engineer. This kills me. I’m wasting this very precious time of my young life where I can build companies, generate employment and add a lot of value to the society instead of drawing BMDs. I love doing startups, thats what I’m made for and I’m good at it. So why not ditch it and do what I love full time. Anyways I code almost all the time of day and night.

But there is more to it. This university has provided so much, I have learned a lot of things from the people who live here and have lived here. And I believe undergraduation is not a process of becoming engineers, its more about unfolding the characters of your personality, figuring out who you want to be and how you are going to achieve it. The person you are and the ideology you follow and person you will shape up to be, its governed a lot by how you spend these four years of undergraduation. Each week I spend here, more I learn about myself and I don’t want to end this shaping up process before it is likely to be. Its a long self realizing journey, it depends on how much I can make out of it.

You make friends for life at this very beautiful place. I don’t want to leave them as of now unless our lives unfolds in a way ( which might happen after we graduate). But moving on is what life is all about, change is something that is permanent rest all is temporary, Still the rational side of my brain is not able to convince me to bail on my friends.

But the biggest reason I don’t want to drop out is I don’t feel ready for it – I just don’t feel I have “to” at this point of time. I have been asked regularly by people of all class – People with 20 years of industry experience, B-school professors, startup CEOs. (Any ways your degree is going to be of no worth to you) . It is of worth to me. Long time after I graduate it will remind me of days when I struggled, juggled between 12 hours of work accompanied with mid-terms and assignments, non stop coding sessions on nightouts and still attending that class with compulsory attendance. The time that made me who I’m now and who I’ll be in future. Learning to manage work and prioritising things. Trying to accommodate work, play, friends and family, failing at times to make it up to all, learning from mistakes and trying no to repeat them. It is what this great place has to offer me and to all the people who live here. Learning my strengths and pushing my limits in all spheres of life.

I will be working at a great organisation, no matter what; but not at the cost of ‘not being able to understand my greatest potential’ and  complement it with ‘greater set of skills’. Life  will be a learning process, It has to be. It will change in its forms from university to real job but it will be an enlightening journey.
Cheers to “Being Hungry and Being Foolish“.

Web scrapping and Automation using PhantomJS

I was recently trying to automate form filling in some sites, searching for options I came across PhantomJS. It is a great library and since I’m just too used to javascript, it turned out to be a very good choice. If you simply want to do simple form filling stuff or parse content from some specific div etc you can try CasperJS. It is a nice little layer on the top of PhantomJS, with simpler function to scrap web content or fill up forms.

So what is the exceptional thing I found about this library. All the tools so far I had used for web scraping – php-curl / python-beautifulsoup / mechanize, all them required figuring out how the form is posted, debugging / figuring out the action of your code was really tough — getting to know from console where you got stuck was really painful. And all of them required lots of code especially php-curl. Some sites would figure out that you are a bot and you get HTTP 403 error.

Now PhantomJS is very exceptional in this regard because what it does is it executes the javascript function you define like it was being executed in the JS console. ( Over the time I have become too comfortable with JS, so whenever I get a choice to use JS over other language I would choose it. So is the reason for my love for Node.js) This gives you an edge because now you just have to know how the DOM is made. Earlier there didnt used to be so much JS validation, AJAX content etc. Now forms are complex and next form fields are pulled via ahah/ajax. Unless what you do totally mimics the behavior of user it would be a tough job to scrape the web. PhantomJS comes to the resuce –

1. You dont have to worry about your user agent – it sent as the same of the computer on which your script runs and still you can modify it if required.

2. PhantomJS uses QtWebkit, so your scripts behavior is same as that of a user trying to access contents from a browser.

3. You can simple trigger events — all of them (click, change, keyup etc). This ensures that if content is fetched via AJAX – you are able to see it.

4. Debugging code is really easy. You can capture the screenshot of the browser (QtWebkit) and see upto what steps your script has accomplished its work. Still you can output stuff to console in fancy colors using CasperJS’s colorize API.

5. There are event listeners almost all of which you could imagine, i you want a custom event you have the liberty to define one. Your script has got options to wait till that click you triggered fetched that content block and not simply keep on running step by step with arbitrary timed sleep/wait.

Btw, CasperJS is widely used to write test cases for web application these days.  You should definitely give it a try. And if you are looking for an extremely powerful web scrapping and automation tool, you should head to Selenium. It is by far the most powerful tool I have come across but remember it has a huge memory heap, it will totally consume your server.