There is no such thing as objective morality, and don't try to tell me you were talking about philosophy when its obvious you're talking about philosophical morality. What "should" be done is always subjective because it begins what one "feels" it should be.
>>thinking about constructing a solid definition for "intelligence", then think about how you might program a computer to posess this quality, and what use it would be.<<
This is a very BAD BAD way to go. Instead work on making it self-aware. Consciousness has NOTHING to do with intelligence. Many people consider me several times more intelligent than most of my peers, does that make me more conscious than them?