Attacks often exploit vulnerabilities in memory to compromise a system or get classified information. In spite of extensive research, attackers are still able to find security holes. In order to address the attacks exploiting vulnerabilities in memory, we propose fine grain dynamic memory access control with profiling. This technique builds profile data for memory accesses with training, and this data is referenced later to check if an access is an allowed legitimate one. To facilitate the fine grain memory access control at a reasonable overhead, instructions and memory words form access group to represent allowed accesses: within one access group, one instruction is accessing at least two memory words or one memory word is accessed by at least two instructions. One access group is assigned with its unique color, and this color is referenced to verify legitimacy of a memory access. In order to handle memory accesses to run-time generated object, we suggest an efficient addressing methods and identifier for associating group information with the object in profiling procedure. To verify its feasibility in statistical point of view, we have implemented our idea in Bochs simulator and results show that memory access control with profiling data can be reliable.