Some time on your development CEPH cluster, ceph monitor may have dhcp address assigned. Any change in the address on ceph-monitor will make the entire cluster unstable. It's easy to change osd ip address and bring the cluster back in healthy state.
It's not true in case of ceph-monitor. One may observe that ceph-mon is running when doing the process grep of ceph-mon,
$ root@ceph1: ceph -s
2015-08-14 10:31:02.483960 7f50601b2700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f505c026050 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f505c0230d0).fault
2015-08-14 10:31:05.484254 7f50582e6700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f5050000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5050004ea0).fault
Hmm... ceph-mon ip address changed to 192.168.1.7
Extract the content of monmap
$root@ceph1: ceph mon getmap -o /tmp/monmap
2015-08-14 10:31:02.483960 7f50601b2700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f505c026050 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f505c0230d0).fault
2015-08-14 10:31:05.484254 7f50582e6700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f5050000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5050004ea0).fault
There is no quorum, since I have only single ceph mon running on my small cluster.
$root@ceph1: stop ceph-mon-all
Extract the monmap directly, if no quorum
$root@ceph1: ceph-mon -i ceph1 --extract-monmap /tmp/monmap
$ root@ceph1:~# monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 1
fsid 3a04ee63-2160-444f-8477-c5235764e349
last_changed 0.000000
created 0.000000
0: 192.168.1.5:6789/0 mon.ceph1
$root@ceph1:~# monmaptool --rm ceph1 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: removing ceph1
monmaptool: writing epoch 1 to /tmp/monmap (0 monitors)
$root@ceph1:~# monmaptool --add ceph1 192.168.1.7:6789 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: writing epoch 1 to /tmp/monmap (1 monitors)
$root@ceph1:~# monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 1
fsid 3a04ee63-2160-444f-8477-c5235764e349
last_changed 0.000000
created 0.000000
0: 192.168.1.7:6789/0 mon.a
root@ceph1:~# ceph-mon -i ceph1 --inject-monmap /tmp/monmap
root@ceph1:~# start ceph-mon-all
Modify /etc/ceph/ceph.conf to reflect the ceph mon ip address as well.
root@ceph1:~# start ceph-osd-all
root@ceph1:~# ceph -s
cluster 3a04ee63-2160-444f-8477-c5235764e349
health HEALTH_OK
monmap e3: 1 mons at {ceph1=192.168.1.7:6789/0}
election epoch 1, quorum 0 ceph1
osdmap e14: 1 osds: 1 up, 1 in
pgmap v22: 128 pgs, 2 pools, 0 bytes data, 0 objects
36404 kB used, 460 GB / 460 GB avail
128 active+clean
We are back online....
It's not true in case of ceph-monitor. One may observe that ceph-mon is running when doing the process grep of ceph-mon,
$ root@ceph1: ceph -s
2015-08-14 10:31:02.483960 7f50601b2700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f505c026050 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f505c0230d0).fault
2015-08-14 10:31:05.484254 7f50582e6700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f5050000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5050004ea0).fault
Hmm... ceph-mon ip address changed to 192.168.1.7
Extract the content of monmap
$root@ceph1: ceph mon getmap -o /tmp/monmap
2015-08-14 10:31:02.483960 7f50601b2700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f505c026050 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f505c0230d0).fault
2015-08-14 10:31:05.484254 7f50582e6700 0 -- :/1010568 >> 192.168.1.5:6789/0 pipe(0x7f5050000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5050004ea0).fault
There is no quorum, since I have only single ceph mon running on my small cluster.
$root@ceph1: stop ceph-mon-all
Extract the monmap directly, if no quorum
$root@ceph1: ceph-mon -i ceph1 --extract-monmap /tmp/monmap
$ root@ceph1:~# monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 1
fsid 3a04ee63-2160-444f-8477-c5235764e349
last_changed 0.000000
created 0.000000
0: 192.168.1.5:6789/0 mon.ceph1
$root@ceph1:~# monmaptool --rm ceph1 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: removing ceph1
monmaptool: writing epoch 1 to /tmp/monmap (0 monitors)
$root@ceph1:~# monmaptool --add ceph1 192.168.1.7:6789 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: writing epoch 1 to /tmp/monmap (1 monitors)
$root@ceph1:~# monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 1
fsid 3a04ee63-2160-444f-8477-c5235764e349
last_changed 0.000000
created 0.000000
0: 192.168.1.7:6789/0 mon.a
root@ceph1:~# ceph-mon -i ceph1 --inject-monmap /tmp/monmap
root@ceph1:~# start ceph-mon-all
Modify /etc/ceph/ceph.conf to reflect the ceph mon ip address as well.
root@ceph1:~# start ceph-osd-all
root@ceph1:~# ceph -s
cluster 3a04ee63-2160-444f-8477-c5235764e349
health HEALTH_OK
monmap e3: 1 mons at {ceph1=192.168.1.7:6789/0}
election epoch 1, quorum 0 ceph1
osdmap e14: 1 osds: 1 up, 1 in
pgmap v22: 128 pgs, 2 pools, 0 bytes data, 0 objects
36404 kB used, 460 GB / 460 GB avail
128 active+clean
We are back online....