Advanced researches on the variational quantum algorithms are actively conducted. In particular, the quantum approximate optimization algorithm (QAOA) is one of the promising variational quantum algorithms and can be applied to various graph-based problems, and is a promising algorithm that shows good performance even in small quantum computers. As is widely known, QAOA obtains the approximate solution via the expectation value of the cost Hamiltonian on the parameterized state. Therefore, in addition to finding the optimal parameters, the proper design of the cost Hamiltonian is important. This paper designs the cost function of the combinatorial optimization problem via Boolean function and maps it to the proper cost Hamiltonian. The proposed cost Hamiltonian design method is applied to the maximum independent set (MIS) and minimum dominating set (MDS) problems.